[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:28:57 dist-test-slave-h6hf krb5kdc[28295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:28:57 dist-test-slave-h6hf krb5kdc[28295](info): set up 2 sockets
Sep 25 01:28:57 dist-test-slave-h6hf krb5kdc[28295](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 01:28:58.667485 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:28:58 dist-test-slave-h6hf krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:28:58.846653 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44849
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:44849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:59.234383 28308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:59.234838 28308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:59.235093 28308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:59.244031 28308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:28:59.244199 28308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:59.244321 28308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:59.244428 28308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:28:59.244518 28308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:28:59.255517 28308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:44849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:59.257220 28308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:59.258844 28308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:59.276470 28313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:59.276875 28314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:59.277256 28316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:59.278825 28308 server_base.cc:1034] running on GCE node
I20260925 01:28:59.280041 28308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:59.281755 28308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:59.282996 28308 hybrid_clock.cc:648] HybridClock initialized: now 1790299739282908 us; error 88 us; skew 500 ppm
I20260925 01:28:59.283366 28308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:59 dist-test-slave-h6hf krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299739, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:59.286775 28308 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:28:59.288524 28308 webserver.cc:460] Webserver started at http://127.27.160.62:39553/ using document root <none> and password file <none>
I20260925 01:28:59.289331 28308 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:59.289479 28308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:59.289803 28308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:59.292838 28308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "abae36d380e649adb7aff2f19c4d60c7"
format_stamp: "Formatted at 2026-09-25 01:28:59 on dist-test-slave-h6hf"
server_key: "8132dc2c7e6af23451badd20272a44f1"
server_key_iv: "bee3d55a4eaa5ab33bf09982853284eb"
server_key_version: "encryptionkey@0"
I20260925 01:28:59.293699 28308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "abae36d380e649adb7aff2f19c4d60c7"
format_stamp: "Formatted at 2026-09-25 01:28:59 on dist-test-slave-h6hf"
server_key: "8132dc2c7e6af23451badd20272a44f1"
server_key_iv: "bee3d55a4eaa5ab33bf09982853284eb"
server_key_version: "encryptionkey@0"
I20260925 01:28:59.298987 28308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:28:59.302744 28323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:59.304303 28308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 01:28:59.304518 28308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "abae36d380e649adb7aff2f19c4d60c7"
format_stamp: "Formatted at 2026-09-25 01:28:59 on dist-test-slave-h6hf"
server_key: "8132dc2c7e6af23451badd20272a44f1"
server_key_iv: "bee3d55a4eaa5ab33bf09982853284eb"
server_key_version: "encryptionkey@0"
I20260925 01:28:59.304744 28308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:28:59.338107 28308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:28:59.343178 28308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:28:59.343492 28308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:28:59.363641 28308 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:44849
I20260925 01:28:59.363737 28374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:44849 every 8 connection(s)
I20260925 01:28:59.365092 28308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:28:59.367664 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 28308
I20260925 01:28:59.368062 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:28:59.370817 28375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:28:59.371090 28288 external_mini_cluster.cc:1428] Setting key ab18f6065440d81e7b90f70a0d006edb
I20260925 01:28:59.383338 28375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7: Bootstrap starting.
I20260925 01:28:59.386659 28375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7: Neither blocks nor log segments found. Creating new log.
I20260925 01:28:59.388695 28375 log.cc:824] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7: Log is configured to *not* fsync() on all Append() calls
I20260925 01:28:59.392421 28375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7: No bootstrap required, opened a new log
I20260925 01:28:59.402057 28375 raft_consensus.cc:348] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abae36d380e649adb7aff2f19c4d60c7" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44849 } }
I20260925 01:28:59.402436 28375 raft_consensus.cc:374] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:28:59.402593 28375 raft_consensus.cc:729] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abae36d380e649adb7aff2f19c4d60c7, State: Initialized, Role: FOLLOWER
I20260925 01:28:59.403241 28375 consensus_queue.cc:260] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abae36d380e649adb7aff2f19c4d60c7" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44849 } }
I20260925 01:28:59.403472 28375 raft_consensus.cc:388] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:28:59.403653 28375 raft_consensus.cc:482] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:28:59.403826 28375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 01:28:59 dist-test-slave-h6hf krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:28:59.407037 28375 raft_consensus.cc:504] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abae36d380e649adb7aff2f19c4d60c7" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44849 } }
I20260925 01:28:59.407743 28375 leader_election.cc:302] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abae36d380e649adb7aff2f19c4d60c7; no voters:
I20260925 01:28:59.408922 28375 leader_election.cc:288] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:28:59.409379 28380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:28:59.411312 28380 raft_consensus.cc:686] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [term 1 LEADER]: Becoming Leader. State: Replica: abae36d380e649adb7aff2f19c4d60c7, State: Running, Role: LEADER
I20260925 01:28:59.411779 28375 sys_catalog.cc:564] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:28:59.411918 28380 consensus_queue.cc:237] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abae36d380e649adb7aff2f19c4d60c7" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44849 } }
I20260925 01:28:59.417297 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:28:59.376031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.27.160.62:44849)
0925 01:28:59.377147 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0925 01:28:59.377164 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:28:59.378622 (+ 1458us) server_negotiation.cc:407] Connection header received
0925 01:28:59.388479 (+ 9857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:28:59.388507 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:28:59.389188 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:28:59.389730 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:28:59.393692 (+ 3962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:59.396266 (+ 2574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:59.398310 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:28:59.399393 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:28:59.399611 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:28:59.406085 (+ 6474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:28:59.406100 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:28:59.406123 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:28:59.406180 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:28:59.410865 (+ 4685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:59.412133 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:59.412143 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:59.412153 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:59.412363 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:28:59.412968 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:28:59.412973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:28:59.412977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:28:59.413931 (+ 954us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:28:59.414257 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 01:28:59.415367 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0925 01:28:59.415798 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":527,"threads_started":1}
I20260925 01:28:59.418124 28381 sys_catalog.cc:455] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abae36d380e649adb7aff2f19c4d60c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abae36d380e649adb7aff2f19c4d60c7" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44849 } } }
I20260925 01:28:59.418596 28381 sys_catalog.cc:458] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:59.420444 28382 sys_catalog.cc:455] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abae36d380e649adb7aff2f19c4d60c7. Latest consensus state: current_term: 1 leader_uuid: "abae36d380e649adb7aff2f19c4d60c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abae36d380e649adb7aff2f19c4d60c7" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44849 } } }
I20260925 01:28:59.421185 28382 sys_catalog.cc:458] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7 [sys.catalog]: This master's current role is: LEADER
I20260925 01:28:59.435890 28389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:28:59.442762 28389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:28:59.455081 28389 catalog_manager.cc:1347] Generated new cluster ID: 59c87ef94d484c42b3770b1f19091e20
I20260925 01:28:59.455215 28389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:28:59.468868 28389 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:28:59.470813 28389 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:28:59.483611 28389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abae36d380e649adb7aff2f19c4d60c7: Generated new TSK 0
I20260925 01:28:59.484962 28389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:28:59.549933 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44849
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:28:59.928377 28402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:28:59.928706 28402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:28:59.928866 28402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:28:59.936429 28402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:28:59.936589 28402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:28:59.936780 28402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:28:59.946964 28402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:28:59.948403 28402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:28:59.949793 28402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:28:59.963342 28407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:59.963266 28410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:28:59.963305 28408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:28:59.964630 28402 server_base.cc:1034] running on GCE node
I20260925 01:28:59.965327 28402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:28:59.966301 28402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:28:59.967510 28402 hybrid_clock.cc:648] HybridClock initialized: now 1790299739967440 us; error 60 us; skew 500 ppm
I20260925 01:28:59.967898 28402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:28:59 dist-test-slave-h6hf krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299739, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:28:59.971309 28402 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:28:59.972783 28402 webserver.cc:460] Webserver started at http://127.27.160.1:40879/ using document root <none> and password file <none>
I20260925 01:28:59.973719 28402 fs_manager.cc:359] Metadata directory not provided
I20260925 01:28:59.973886 28402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:28:59.974242 28402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:28:59.977722 28402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "ac636c77a4d947b5af4544bfcb24e825"
format_stamp: "Formatted at 2026-09-25 01:28:59 on dist-test-slave-h6hf"
server_key: "765d2e49a502829e57bc86aa9984bd13"
server_key_iv: "4ececc8f904318c71f088832dd33d4e6"
server_key_version: "encryptionkey@0"
I20260925 01:28:59.978638 28402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "ac636c77a4d947b5af4544bfcb24e825"
format_stamp: "Formatted at 2026-09-25 01:28:59 on dist-test-slave-h6hf"
server_key: "765d2e49a502829e57bc86aa9984bd13"
server_key_iv: "4ececc8f904318c71f088832dd33d4e6"
server_key_version: "encryptionkey@0"
I20260925 01:28:59.984839 28402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 01:28:59.989035 28417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:28:59.990593 28402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:28:59.990756 28402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "ac636c77a4d947b5af4544bfcb24e825"
format_stamp: "Formatted at 2026-09-25 01:28:59 on dist-test-slave-h6hf"
server_key: "765d2e49a502829e57bc86aa9984bd13"
server_key_iv: "4ececc8f904318c71f088832dd33d4e6"
server_key_version: "encryptionkey@0"
I20260925 01:28:59.991026 28402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:00.006795 28402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:00.013187 28402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:00.013573 28402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:00.014827 28402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:00.016366 28402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:00.016440 28402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:00.016537 28402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:00.016590 28402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:00.050514 28402 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:35013
I20260925 01:29:00.050642 28529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:35013 every 8 connection(s)
I20260925 01:29:00.051842 28402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:00.060173 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 28402
I20260925 01:29:00.060428 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:00.061035 28288 external_mini_cluster.cc:1428] Setting key 5c7704638f28a8b47d96ac80b3ae9739
Sep 25 01:29:00 dist-test-slave-h6hf krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299739, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:00.074513 28532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:00.054695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:41403 (local address 127.27.160.62:44849)
0925 01:29:00.055293 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:00.055300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:00.056277 (+ 977us) server_negotiation.cc:407] Connection header received
0925 01:29:00.058328 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:00.058333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:00.058403 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:00.058507 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:00.060267 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:00.061109 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:00.062936 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:00.063500 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:00.063654 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:00.067196 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:00.067204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:00.067210 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:00.067256 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:00.070751 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:00.071715 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:00.071720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:00.071722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:00.071869 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:00.072419 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:00.072424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:00.072426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:00.072706 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:00.072825 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:00.073809 (+ 984us) server_negotiation.cc:299] Negotiation successful
0925 01:29:00.074009 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":293,"threads_started":1}
I20260925 01:29:00.075515 28533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:00.055210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44849 (local address 127.27.160.1:41403)
0925 01:29:00.056082 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:00.056132 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:00.058027 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:00.058774 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:00.058783 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:00.059458 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:00.060051 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:00.060070 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:00.061302 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:00.061310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:00.062763 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:00.062774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:00.063717 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:00.063722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:00.063918 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:00.064689 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:00.064713 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:00.066956 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:00.070904 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:00.070916 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:00.070937 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:00.071574 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:00.072008 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:00.072015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:00.072022 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:00.072277 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:00.072845 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:00.072855 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:00.073372 (+ 517us) client_negotiation.cc:769] Sending connection context
0925 01:29:00.073743 (+ 371us) client_negotiation.cc:240] Negotiation successful
0925 01:29:00.074311 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":318,"threads_started":1}
I20260925 01:29:00.077178 28530 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44849
I20260925 01:29:00.077494 28530 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:29:00.078254 28530 heartbeater.cc:511] Master 127.27.160.62:44849 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
I20260925 01:29:00.080581 28340 ts_manager.cc:194] Registered new tserver with Master: ac636c77a4d947b5af4544bfcb24e825 (127.27.160.1:35013)
I20260925 01:29:00.082136 28340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:41403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:00.104931 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44849
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:00.575410 28536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:00.575770 28536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:00.575923 28536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:00.586388 28536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:00.586570 28536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:00.586719 28536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:00.596836 28536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:00.598232 28536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:00.599493 28536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:00.612831 28544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:00.613072 28541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:00.613039 28542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:00.613780 28536 server_base.cc:1034] running on GCE node
I20260925 01:29:00.614441 28536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:00.615453 28536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:00.616693 28536 hybrid_clock.cc:648] HybridClock initialized: now 1790299740616660 us; error 32 us; skew 500 ppm
I20260925 01:29:00.617041 28536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:00 dist-test-slave-h6hf krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299740, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:00.620316 28536 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:00.621933 28536 webserver.cc:460] Webserver started at http://127.27.160.2:39197/ using document root <none> and password file <none>
I20260925 01:29:00.622639 28536 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:00.622785 28536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:00.623109 28536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:00.626044 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "4e6362eea9db4e688a7697ded7c939fa"
format_stamp: "Formatted at 2026-09-25 01:29:00 on dist-test-slave-h6hf"
server_key: "7c15067a33a3e0650d0baa10558e04a6"
server_key_iv: "017b1e917ec611060d31608bd31a28d4"
server_key_version: "encryptionkey@0"
I20260925 01:29:00.626806 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "4e6362eea9db4e688a7697ded7c939fa"
format_stamp: "Formatted at 2026-09-25 01:29:00 on dist-test-slave-h6hf"
server_key: "7c15067a33a3e0650d0baa10558e04a6"
server_key_iv: "017b1e917ec611060d31608bd31a28d4"
server_key_version: "encryptionkey@0"
I20260925 01:29:00.632216 28536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:29:00.636040 28551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:00.637693 28536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:00.637923 28536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "4e6362eea9db4e688a7697ded7c939fa"
format_stamp: "Formatted at 2026-09-25 01:29:00 on dist-test-slave-h6hf"
server_key: "7c15067a33a3e0650d0baa10558e04a6"
server_key_iv: "017b1e917ec611060d31608bd31a28d4"
server_key_version: "encryptionkey@0"
I20260925 01:29:00.638202 28536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:00.659477 28536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:00.665566 28536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:00.666008 28536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:00.667316 28536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:00.669543 28536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:00.669696 28536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:00.669916 28536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:00.670042 28536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:00.722209 28536 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:34167
I20260925 01:29:00.722245 28663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:34167 every 8 connection(s)
I20260925 01:29:00.723615 28536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:00.726840 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 28536
I20260925 01:29:00.727094 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:00.727797 28288 external_mini_cluster.cc:1428] Setting key 563f2c501989ca4f2721803a7fa42e8c
Sep 25 01:29:00 dist-test-slave-h6hf krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299740, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
I20260925 01:29:00.756093 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:00.727423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:36905 (local address 127.27.160.62:44849)
0925 01:29:00.733698 (+ 6275us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:00.733703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:00.733724 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:29:00.734929 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:00.734935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:00.735019 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:00.735162 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:00.738392 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:00.739273 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:00.742229 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:00.742947 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:00.743060 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:00.748917 (+ 5857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:00.748923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:00.748927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:00.748963 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:00.752296 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:00.753230 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:00.753235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:00.753237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:00.753363 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:00.753916 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:00.753920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:00.753922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:00.754478 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:00.754673 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:00.755651 (+ 978us) server_negotiation.cc:299] Negotiation successful
0925 01:29:00.755832 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6128,"thread_start_us":6000,"threads_started":1}
I20260925 01:29:00.758179 28667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:00.728290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44849 (local address 127.27.160.2:36905)
0925 01:29:00.730142 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:00.730220 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:00.734539 (+ 4319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:00.735607 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:00.735623 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:00.736826 (+ 1203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:00.738162 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:00.738199 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:00.739545 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:00.739554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:00.741975 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:00.741988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:00.743350 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:00.743360 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:00.743788 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:00.744742 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:00.744808 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:00.748575 (+ 3767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:00.752454 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:00.752464 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:00.752485 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:00.753024 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:00.753495 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:00.753500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:00.753504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:00.753782 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:00.754653 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:00.754662 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:00.755185 (+ 523us) client_negotiation.cc:769] Sending connection context
0925 01:29:00.755599 (+ 414us) client_negotiation.cc:240] Negotiation successful
0925 01:29:00.756300 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"thread_start_us":952,"threads_started":1}
I20260925 01:29:00.759812 28664 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44849
I20260925 01:29:00.760177 28664 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:00.761066 28664 heartbeater.cc:511] Master 127.27.160.62:44849 requested a full tablet report, sending...
I20260925 01:29:00.762928 28340 ts_manager.cc:194] Registered new tserver with Master: 4e6362eea9db4e688a7697ded7c939fa (127.27.160.2:34167)
I20260925 01:29:00.764464 28340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:36905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:00.780143 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44849
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:29:01.085088 28530 heartbeater.cc:503] Master 127.27.160.62:44849 was elected leader, sending a full tablet report...
W20260925 01:29:01.210469 28670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:01.210892 28670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:01.211145 28670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:01.221752 28670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:01.221992 28670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:01.222237 28670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:01.234589 28670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:01.236074 28670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:01.237710 28670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:01.252569 28675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:01.252679 28676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:01.252628 28678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:01.253796 28670 server_base.cc:1034] running on GCE node
I20260925 01:29:01.254431 28670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:01.255435 28670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:01.256672 28670 hybrid_clock.cc:648] HybridClock initialized: now 1790299741256586 us; error 67 us; skew 500 ppm
I20260925 01:29:01.257033 28670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:01 dist-test-slave-h6hf krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299741, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:01.260356 28670 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:01.261895 28670 webserver.cc:460] Webserver started at http://127.27.160.3:33041/ using document root <none> and password file <none>
I20260925 01:29:01.262602 28670 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:01.262735 28670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:01.263067 28670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:01.266906 28670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "016377df1e6643768adb8607783fede6"
format_stamp: "Formatted at 2026-09-25 01:29:01 on dist-test-slave-h6hf"
server_key: "ebc0b3dca79789e9dd73dcb2b4b9b28d"
server_key_iv: "49ef4a2f9acef507dfcf67f9b1d29413"
server_key_version: "encryptionkey@0"
I20260925 01:29:01.267807 28670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "016377df1e6643768adb8607783fede6"
format_stamp: "Formatted at 2026-09-25 01:29:01 on dist-test-slave-h6hf"
server_key: "ebc0b3dca79789e9dd73dcb2b4b9b28d"
server_key_iv: "49ef4a2f9acef507dfcf67f9b1d29413"
server_key_version: "encryptionkey@0"
I20260925 01:29:01.273855 28670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 01:29:01.278445 28685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:01.279989 28670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260925 01:29:01.280269 28670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "016377df1e6643768adb8607783fede6"
format_stamp: "Formatted at 2026-09-25 01:29:01 on dist-test-slave-h6hf"
server_key: "ebc0b3dca79789e9dd73dcb2b4b9b28d"
server_key_iv: "49ef4a2f9acef507dfcf67f9b1d29413"
server_key_version: "encryptionkey@0"
I20260925 01:29:01.280540 28670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:01.296684 28670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:01.302934 28670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:01.303423 28670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:01.304999 28670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:01.306852 28670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:01.307030 28670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:01.307245 28670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:01.307372 28670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:01.362608 28670 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:46043
I20260925 01:29:01.362699 28797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:46043 every 8 connection(s)
I20260925 01:29:01.364023 28670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:01.370287 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 28670
I20260925 01:29:01.370486 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:01.370980 28288 external_mini_cluster.cc:1428] Setting key c1ea99f68dbda3c3f759f6989e9398a7
Sep 25 01:29:01 dist-test-slave-h6hf krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299741, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:01.385324 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:01.366819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:34291 (local address 127.27.160.62:44849)
0925 01:29:01.367396 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:01.367400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:01.368400 (+ 1000us) server_negotiation.cc:407] Connection header received
0925 01:29:01.370315 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:01.370321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:01.370429 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:01.370555 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:01.372191 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.373301 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.374922 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.375333 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.375464 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:01.379223 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:01.379228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:01.379231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:01.379266 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:01.382042 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.382922 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.382937 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.382943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.383131 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.383572 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.383576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.383578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.383843 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:01.384025 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:01.384671 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 01:29:01.384867 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":336,"threads_started":1}
I20260925 01:29:01.386556 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:01.367462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44849 (local address 127.27.160.3:34291)
0925 01:29:01.368239 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:01.368276 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:01.370044 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:01.370803 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:01.370812 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:01.371326 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:01.371998 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:01.372026 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.373447 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.373459 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:01.374778 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:01.374791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.375551 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.375557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:01.375792 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:01.376516 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:01.376539 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:01.378995 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:01.382200 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:01.382208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:01.382223 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:01.382716 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:01.383250 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:01.383254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:01.383256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:01.383459 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:01.384006 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:01.384013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:01.384345 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 01:29:01.384594 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 01:29:01.385099 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":303,"threads_started":1}
I20260925 01:29:01.388167 28798 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44849
I20260925 01:29:01.388464 28798 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:01.389221 28798 heartbeater.cc:511] Master 127.27.160.62:44849 requested a full tablet report, sending...
I20260925 01:29:01.390581 28340 ts_manager.cc:194] Registered new tserver with Master: 016377df1e6643768adb8607783fede6 (127.27.160.3:46043)
I20260925 01:29:01.391522 28340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:34291
I20260925 01:29:01.399945 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:29:01 dist-test-slave-h6hf krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:29:01 dist-test-slave-h6hf krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:01.445005 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:01.429446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52896 (local address 127.27.160.1:35013)
0925 01:29:01.430256 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:01.430267 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:01.430305 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:29:01.430383 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:01.430392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:01.430650 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:01.430786 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:01.432159 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.433194 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.434509 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.435077 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.435177 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:01.439159 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:01.439171 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:01.439186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:01.439222 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:01.442428 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.443041 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.443048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.443051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.443176 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.443545 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.443549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.443551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.443895 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:01.444125 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:01.444391 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 01:29:01.444635 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":358,"threads_started":1}
W20260925 01:29:01.446451 28445 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52896
Sep 25 01:29:01 dist-test-slave-h6hf krb5kdc[28295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:29:01 dist-test-slave-h6hf krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:01.496043 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:01.477481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.27.160.1:35013)
0925 01:29:01.477688 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:01.477710 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:01.478178 (+ 468us) server_negotiation.cc:407] Connection header received
0925 01:29:01.478391 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:01.478403 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:01.478468 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:01.478576 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:01.480588 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.481455 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.483003 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.483706 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.483861 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:01.489296 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:01.489302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:01.489305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:01.489343 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:01.492802 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.493699 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.493709 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.493712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.493895 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.494310 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.494319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.494322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.494602 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:01.494920 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:01.495286 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 01:29:01.495568 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:29:01.498135 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 28402
I20260925 01:29:01.513728 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:01.499540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.27.160.1:35013)
0925 01:29:01.499793 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:01.499801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:01.499960 (+ 159us) server_negotiation.cc:407] Connection header received
0925 01:29:01.500182 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:01.500191 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:01.500263 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:01.500382 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:01.501966 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.502777 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.504334 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.504831 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.504995 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:01.507782 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:01.507788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:01.507792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:01.507828 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:01.510912 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.511641 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.511654 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.511656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.511799 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.512228 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.512236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.512240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.512511 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:01.512776 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:01.513102 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 01:29:01.513348 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260925 01:29:01.514707 28525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:01.602560 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 28402
I20260925 01:29:01.621824 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 28536
Sep 25 01:29:01 dist-test-slave-h6hf krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:29:01.639242 28813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:01.622795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.27.160.2:34167)
0925 01:29:01.623548 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:01.623555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:01.623580 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:29:01.623646 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:01.623652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:01.623968 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:01.624186 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:01.625504 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.626507 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.627866 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.628591 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.628739 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:01.633195 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:01.633208 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:01.633224 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:01.633285 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:01.636369 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.637062 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.637069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.637072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.637299 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.637663 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.637667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.637669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.638087 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:01.638324 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:01.638589 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 01:29:01.638791 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":450,"threads_started":1}
I20260925 01:29:01.640658 28659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:01.724550 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 28536
I20260925 01:29:01.747028 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 28670
Sep 25 01:29:01 dist-test-slave-h6hf krb5kdc[28295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:29:01.768826 28815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:01.748151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.27.160.3:46043)
0925 01:29:01.748749 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:01.748755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:01.748784 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:29:01.748844 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:01.748851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:01.749059 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:01.749259 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:01.750426 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.751435 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.752733 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:01.753397 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:01.753496 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:01.763120 (+ 9624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:01.763136 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:01.763153 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:01.763196 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:01.766144 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.766841 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.766848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.766850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.766967 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:01.767318 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:01.767321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:01.767323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:01.767662 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:01.767888 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:01.768213 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 01:29:01.768409 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":292,"threads_started":1}
I20260925 01:29:01.770226 28793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:01.855525 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 28670
I20260925 01:29:01.876314 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 28308
I20260925 01:29:01.877925 28370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:01.993374 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 28308
2026-09-25T01:29:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4426 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:02 dist-test-slave-h6hf krb5kdc[28820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:02 dist-test-slave-h6hf krb5kdc[28820](info): set up 2 sockets
Sep 25 01:29:02 dist-test-slave-h6hf krb5kdc[28820](info): commencing operation
krb5kdc: starting...
W20260925 01:29:03.078064 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:03 dist-test-slave-h6hf krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:03.265231 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:42797
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:03.599987 28833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:03.600299 28833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:03.600461 28833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:03.607731 28833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:03.607851 28833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:03.607961 28833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:03.608076 28833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:03.608158 28833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:03.618180 28833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:42797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:03.619568 28833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:03.620973 28833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:03.635239 28839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:03.635267 28838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:03.635969 28841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:03.637316 28833 server_base.cc:1034] running on GCE node
I20260925 01:29:03.638160 28833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:03.639722 28833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:03.640926 28833 hybrid_clock.cc:648] HybridClock initialized: now 1790299743640833 us; error 88 us; skew 500 ppm
I20260925 01:29:03.641320 28833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:03 dist-test-slave-h6hf krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299743, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:03.649657 28833 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:03.651283 28833 webserver.cc:460] Webserver started at http://127.27.160.62:33499/ using document root <none> and password file <none>
I20260925 01:29:03.652053 28833 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:03.652164 28833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:03.652467 28833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:03.655884 28833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "12de860ce1ed4827bfa680363975757f"
format_stamp: "Formatted at 2026-09-25 01:29:03 on dist-test-slave-h6hf"
server_key: "04c261902861ded99024fb740d48e742"
server_key_iv: "e0e00951242bc3c867f764d6282ac3cc"
server_key_version: "encryptionkey@0"
I20260925 01:29:03.656656 28833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "12de860ce1ed4827bfa680363975757f"
format_stamp: "Formatted at 2026-09-25 01:29:03 on dist-test-slave-h6hf"
server_key: "04c261902861ded99024fb740d48e742"
server_key_iv: "e0e00951242bc3c867f764d6282ac3cc"
server_key_version: "encryptionkey@0"
I20260925 01:29:03.662412 28833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:29:03.666487 28848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:03.668155 28833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:03.668493 28833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "12de860ce1ed4827bfa680363975757f"
format_stamp: "Formatted at 2026-09-25 01:29:03 on dist-test-slave-h6hf"
server_key: "04c261902861ded99024fb740d48e742"
server_key_iv: "e0e00951242bc3c867f764d6282ac3cc"
server_key_version: "encryptionkey@0"
I20260925 01:29:03.668712 28833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:03.685349 28833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:03.690024 28833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:03.690323 28833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:03.709088 28833 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:42797
I20260925 01:29:03.709100 28899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:42797 every 8 connection(s)
I20260925 01:29:03.710593 28833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:03.714241 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 28833
I20260925 01:29:03.714502 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:03.714541 28900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:03.715175 28288 external_mini_cluster.cc:1428] Setting key 2ee84bba024bf4f3ba0ed15e2762cd68
I20260925 01:29:03.729183 28900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f: Bootstrap starting.
Sep 25 01:29:03 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:03.732345 28900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:03.733773 28900 log.cc:824] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:03.737576 28900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f: No bootstrap required, opened a new log
I20260925 01:29:03.742063 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:03.717216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.27.160.62:42797)
0925 01:29:03.718129 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:03.718144 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:03.718197 (+ 53us) server_negotiation.cc:407] Connection header received
0925 01:29:03.719932 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:03.719968 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:03.720510 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:03.720999 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:03.722730 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:03.724644 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:03.726362 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:03.727465 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:03.727653 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:03.732574 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:03.732586 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:03.732608 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:03.732667 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:03.736897 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:03.737785 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:03.737794 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:03.737802 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:03.738032 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:03.738460 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:03.738468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:03.738471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:03.739177 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:03.739485 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:03.740029 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 01:29:03.740503 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":396,"threads_started":1}
I20260925 01:29:03.747323 28900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12de860ce1ed4827bfa680363975757f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42797 } }
I20260925 01:29:03.747663 28900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:03.747767 28900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12de860ce1ed4827bfa680363975757f, State: Initialized, Role: FOLLOWER
I20260925 01:29:03.748402 28900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12de860ce1ed4827bfa680363975757f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42797 } }
I20260925 01:29:03.748652 28900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:03.748821 28900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:03.748991 28900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:03.751140 28900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12de860ce1ed4827bfa680363975757f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42797 } }
I20260925 01:29:03.751653 28900 leader_election.cc:302] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12de860ce1ed4827bfa680363975757f; no voters:
I20260925 01:29:03.752323 28900 leader_election.cc:288] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:03.752478 28905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:03.753283 28905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [term 1 LEADER]: Becoming Leader. State: Replica: 12de860ce1ed4827bfa680363975757f, State: Running, Role: LEADER
I20260925 01:29:03.753942 28905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12de860ce1ed4827bfa680363975757f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42797 } }
I20260925 01:29:03.754659 28900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:03.757371 28907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12de860ce1ed4827bfa680363975757f. Latest consensus state: current_term: 1 leader_uuid: "12de860ce1ed4827bfa680363975757f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12de860ce1ed4827bfa680363975757f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42797 } } }
I20260925 01:29:03.757539 28906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12de860ce1ed4827bfa680363975757f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12de860ce1ed4827bfa680363975757f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42797 } } }
I20260925 01:29:03.757813 28907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:03.757813 28906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:03.759866 28914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:03.765743 28914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:03.775259 28914 catalog_manager.cc:1347] Generated new cluster ID: 58db3b3671994102aced553192dd4035
I20260925 01:29:03.775382 28914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:03.811249 28914 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:03.812827 28914 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:03.823128 28914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12de860ce1ed4827bfa680363975757f: Generated new TSK 0
I20260925 01:29:03.824084 28914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:29:03.877849 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42797
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:04.212878 28926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:04.213248 28926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:04.213366 28926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:04.220639 28926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:04.220764 28926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:04.220882 28926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:04.230983 28926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:04.232259 28926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:04.233611 28926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:04.248005 28931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:04.248101 28934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:04.248121 28932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:04.248929 28926 server_base.cc:1034] running on GCE node
I20260925 01:29:04.249699 28926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:04.250730 28926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:04.252008 28926 hybrid_clock.cc:648] HybridClock initialized: now 1790299744251948 us; error 61 us; skew 500 ppm
I20260925 01:29:04.252357 28926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:04 dist-test-slave-h6hf krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299744, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:04.255985 28926 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:29:04.257625 28926 webserver.cc:460] Webserver started at http://127.27.160.1:36873/ using document root <none> and password file <none>
I20260925 01:29:04.258317 28926 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:04.258432 28926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:04.258724 28926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:04.262056 28926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "8409e67458d2436ab7d2520fbd58b53b"
format_stamp: "Formatted at 2026-09-25 01:29:04 on dist-test-slave-h6hf"
server_key: "eb4497af6da943e689167c2414548044"
server_key_iv: "1fe334307d8bfea1c290209ba8d30c4d"
server_key_version: "encryptionkey@0"
I20260925 01:29:04.262872 28926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "8409e67458d2436ab7d2520fbd58b53b"
format_stamp: "Formatted at 2026-09-25 01:29:04 on dist-test-slave-h6hf"
server_key: "eb4497af6da943e689167c2414548044"
server_key_iv: "1fe334307d8bfea1c290209ba8d30c4d"
server_key_version: "encryptionkey@0"
I20260925 01:29:04.268672 28926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:29:04.272629 28941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:04.274251 28926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:04.274464 28926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "8409e67458d2436ab7d2520fbd58b53b"
format_stamp: "Formatted at 2026-09-25 01:29:04 on dist-test-slave-h6hf"
server_key: "eb4497af6da943e689167c2414548044"
server_key_iv: "1fe334307d8bfea1c290209ba8d30c4d"
server_key_version: "encryptionkey@0"
I20260925 01:29:04.274722 28926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:04.300232 28926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:04.304538 28926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:04.304798 28926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:04.305828 28926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:04.307163 28926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:04.307236 28926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:04.307318 28926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:04.307361 28926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:04.341282 28926 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:37183
I20260925 01:29:04.341370 29053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:37183 every 8 connection(s)
I20260925 01:29:04.342618 28926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:04.346481 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 28926
I20260925 01:29:04.346700 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:04.347411 28288 external_mini_cluster.cc:1428] Setting key c16ebd85478369cca33c560e3e7eaa6e
Sep 25 01:29:04 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299744, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:29:04.366927 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:04.345299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:42923 (local address 127.27.160.62:42797)
0925 01:29:04.345726 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:04.345731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:04.346999 (+ 1268us) server_negotiation.cc:407] Connection header received
0925 01:29:04.348996 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:04.349002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:04.349088 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:04.349236 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:04.351333 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:04.352024 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:04.354564 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:04.355322 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:04.355462 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:04.359635 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:04.359640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:04.359644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:04.359679 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:04.363119 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:04.364196 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:04.364201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:04.364203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:04.364335 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:04.364895 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:04.364899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:04.364901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:04.365323 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:04.365532 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:04.366379 (+ 847us) server_negotiation.cc:299] Negotiation successful
0925 01:29:04.366631 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":214,"threads_started":1}
I20260925 01:29:04.367924 29057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:04.345893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:42797 (local address 127.27.160.1:42923)
0925 01:29:04.346763 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:04.346812 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:04.348700 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:04.349536 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:04.349548 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:04.350310 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:04.351151 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:04.351181 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:04.352205 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:04.352213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:04.354403 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:04.354421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:04.355578 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:04.355583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:04.355862 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:04.356570 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:04.356593 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:04.359348 (+ 2755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:04.363289 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:04.363299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:04.363320 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:04.364009 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:04.364466 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:04.364474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:04.364481 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:04.364764 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:04.365478 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:04.365490 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:04.365935 (+ 445us) client_negotiation.cc:769] Sending connection context
0925 01:29:04.366172 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 01:29:04.366662 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":306,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
I20260925 01:29:04.369669 29054 heartbeater.cc:347] Connected to a master server at 127.27.160.62:42797
I20260925 01:29:04.370045 29054 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:04.371012 29054 heartbeater.cc:511] Master 127.27.160.62:42797 requested a full tablet report, sending...
I20260925 01:29:04.373849 28865 ts_manager.cc:194] Registered new tserver with Master: 8409e67458d2436ab7d2520fbd58b53b (127.27.160.1:37183)
I20260925 01:29:04.376325 28865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:42923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:04.392824 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42797
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:04.721230 29060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:04.721555 29060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:04.721730 29060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:04.729202 29060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:04.729367 29060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:04.729561 29060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:04.739861 29060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:04.741333 29060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:04.742736 29060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:04.756109 29068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:04.756289 29066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:04.756317 29065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:04.757658 29060 server_base.cc:1034] running on GCE node
I20260925 01:29:04.758319 29060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:04.759359 29060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:04.760558 29060 hybrid_clock.cc:648] HybridClock initialized: now 1790299744760507 us; error 42 us; skew 500 ppm
I20260925 01:29:04.760890 29060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:04 dist-test-slave-h6hf krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299744, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:04.763994 29060 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:04.765632 29060 webserver.cc:460] Webserver started at http://127.27.160.2:36611/ using document root <none> and password file <none>
I20260925 01:29:04.766319 29060 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:04.766450 29060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:04.766772 29060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:04.769704 29060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "dd24bf55a9a147b58a4ed5810d82cc90"
format_stamp: "Formatted at 2026-09-25 01:29:04 on dist-test-slave-h6hf"
server_key: "00ac7f9be904b62264f7b7398279c4fd"
server_key_iv: "238e6f85471ca80491faeac711940011"
server_key_version: "encryptionkey@0"
I20260925 01:29:04.770478 29060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "dd24bf55a9a147b58a4ed5810d82cc90"
format_stamp: "Formatted at 2026-09-25 01:29:04 on dist-test-slave-h6hf"
server_key: "00ac7f9be904b62264f7b7398279c4fd"
server_key_iv: "238e6f85471ca80491faeac711940011"
server_key_version: "encryptionkey@0"
I20260925 01:29:04.775611 29060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:29:04.779261 29075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:04.780633 29060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:29:04.780808 29060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "dd24bf55a9a147b58a4ed5810d82cc90"
format_stamp: "Formatted at 2026-09-25 01:29:04 on dist-test-slave-h6hf"
server_key: "00ac7f9be904b62264f7b7398279c4fd"
server_key_iv: "238e6f85471ca80491faeac711940011"
server_key_version: "encryptionkey@0"
I20260925 01:29:04.781016 29060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:04.800518 29060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:04.804493 29060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:04.804805 29060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:04.805846 29060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:04.807199 29060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:04.807271 29060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:04.807402 29060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:04.807451 29060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:04.841343 29060 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:39341
I20260925 01:29:04.841398 29187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:39341 every 8 connection(s)
I20260925 01:29:04.842672 29060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:04.851017 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 29060
I20260925 01:29:04.851233 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:04.851812 28288 external_mini_cluster.cc:1428] Setting key 2a8655b1c32e9c084edd9d13a853eed7
Sep 25 01:29:04 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299744, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:04.865262 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:04.845430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:53211 (local address 127.27.160.62:42797)
0925 01:29:04.845629 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:04.845633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:04.846736 (+ 1103us) server_negotiation.cc:407] Connection header received
0925 01:29:04.848610 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:04.848616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:04.848708 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:04.848837 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:04.850447 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:04.851231 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:04.853476 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:04.854293 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:04.854461 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:04.858939 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:04.858945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:04.858948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:04.858982 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:04.861961 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:04.862754 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:04.862766 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:04.862770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:04.862898 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:04.863373 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:04.863382 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:04.863384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:04.863669 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:04.863850 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:04.864616 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 01:29:04.864858 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:29:04.866328 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:04.845862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:42797 (local address 127.27.160.2:53211)
0925 01:29:04.846561 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:04.846607 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:04.848324 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:04.849092 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:04.849100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:04.849690 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:04.850264 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:04.850288 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:04.851432 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:04.851438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:04.853312 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:04.853325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:04.854556 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:04.854579 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:04.854880 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:04.855691 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:04.855729 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:04.858647 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:04.862125 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:04.862135 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:04.862156 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:04.862613 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:04.863023 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:04.863028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:04.863031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:04.863244 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:04.863823 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:04.863833 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:04.864226 (+ 393us) client_negotiation.cc:769] Sending connection context
0925 01:29:04.864519 (+ 293us) client_negotiation.cc:240] Negotiation successful
0925 01:29:04.865107 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":261,"threads_started":1}
I20260925 01:29:04.868110 29188 heartbeater.cc:347] Connected to a master server at 127.27.160.62:42797
I20260925 01:29:04.868471 29188 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:29:04.869534 29188 heartbeater.cc:511] Master 127.27.160.62:42797 requested a full tablet report, sending...
I20260925 01:29:04.871413 28865 ts_manager.cc:194] Registered new tserver with Master: dd24bf55a9a147b58a4ed5810d82cc90 (127.27.160.2:39341)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
I20260925 01:29:04.872594 28865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:53211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:04.896857 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42797
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:39477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:05.226627 29193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:05.226907 29193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:05.227092 29193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:05.234304 29193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:05.234422 29193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:05.234639 29193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:05.244733 29193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42797
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:05.246114 29193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:05.247439 29193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:05.260756 29198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:05.260820 29201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:05.260854 29199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:05.261744 29193 server_base.cc:1034] running on GCE node
I20260925 01:29:05.262450 29193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:05.263422 29193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:05.264658 29193 hybrid_clock.cc:648] HybridClock initialized: now 1790299745264608 us; error 48 us; skew 500 ppm
I20260925 01:29:05.265048 29193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:05 dist-test-slave-h6hf krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299745, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:05.268549 29193 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:05.270260 29193 webserver.cc:460] Webserver started at http://127.27.160.3:39177/ using document root <none> and password file <none>
I20260925 01:29:05.270988 29193 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:05.271135 29193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:05.271456 29193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:05.274775 29193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "30606294240f40f8a2d894244b58da53"
format_stamp: "Formatted at 2026-09-25 01:29:05 on dist-test-slave-h6hf"
server_key: "c2bb5e32200e4fec9399161c399239c2"
server_key_iv: "67591aff2b8e3499a37bae2ebfcf9057"
server_key_version: "encryptionkey@0"
I20260925 01:29:05.275647 29193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "30606294240f40f8a2d894244b58da53"
format_stamp: "Formatted at 2026-09-25 01:29:05 on dist-test-slave-h6hf"
server_key: "c2bb5e32200e4fec9399161c399239c2"
server_key_iv: "67591aff2b8e3499a37bae2ebfcf9057"
server_key_version: "encryptionkey@0"
I20260925 01:29:05.281535 29193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:29:05.285099 29208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:05.286625 29193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:05.286821 29193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "30606294240f40f8a2d894244b58da53"
format_stamp: "Formatted at 2026-09-25 01:29:05 on dist-test-slave-h6hf"
server_key: "c2bb5e32200e4fec9399161c399239c2"
server_key_iv: "67591aff2b8e3499a37bae2ebfcf9057"
server_key_version: "encryptionkey@0"
I20260925 01:29:05.287055 29193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:05.317481 29193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:05.321588 29193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:05.321913 29193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:05.322911 29193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:05.324342 29193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:05.324445 29193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:05.324646 29193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:05.324800 29193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:05.360080 29193 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:39285
I20260925 01:29:05.360179 29320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:39285 every 8 connection(s)
I20260925 01:29:05.361469 29193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:05.365028 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 29193
I20260925 01:29:05.365279 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:05.365736 28288 external_mini_cluster.cc:1428] Setting key e89174180a2465c6b9b33c3613b813e8
Sep 25 01:29:05 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299745, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:05.380029 29054 heartbeater.cc:503] Master 127.27.160.62:42797 was elected leader, sending a full tablet report...
I20260925 01:29:05.384181 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.364194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:56737 (local address 127.27.160.62:42797)
0925 01:29:05.364391 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.364395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.365770 (+ 1375us) server_negotiation.cc:407] Connection header received
0925 01:29:05.368031 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.368036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.368130 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.368246 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:05.370214 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.370989 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.372785 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.373331 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.373437 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.377548 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:05.377553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:05.377556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:05.377596 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:05.380805 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.381857 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.381862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.381866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.382007 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.382456 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.382461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.382471 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.382736 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:05.382887 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.383604 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.383824 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:29:05.385339 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.364691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:42797 (local address 127.27.160.3:56737)
0925 01:29:05.365580 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:05.365629 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:05.367728 (+ 2099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:05.368599 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:05.368616 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:05.369410 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:05.370041 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.370060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.371143 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.371147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.372620 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.372634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.373619 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.373626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.373875 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.374756 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:05.374780 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:05.377320 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:05.380986 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:05.380993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:05.381007 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:05.381704 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:05.382131 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:05.382135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:05.382139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:05.382336 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:05.382867 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:05.382875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:05.383226 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 01:29:05.383495 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 01:29:05.384038 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":437,"threads_started":1}
I20260925 01:29:05.386840 29321 heartbeater.cc:347] Connected to a master server at 127.27.160.62:42797
I20260925 01:29:05.387126 29321 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:05.387781 29321 heartbeater.cc:511] Master 127.27.160.62:42797 requested a full tablet report, sending...
I20260925 01:29:05.389235 28864 ts_manager.cc:194] Registered new tserver with Master: 30606294240f40f8a2d894244b58da53 (127.27.160.3:39285)
I20260925 01:29:05.390059 28864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:56737
I20260925 01:29:05.394384 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:29:05.413476 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.400652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.27.160.62:42797)
0925 01:29:05.400829 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.400834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.401447 (+ 613us) server_negotiation.cc:407] Connection header received
0925 01:29:05.401603 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.401607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.401669 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.401758 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:05.403258 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.403967 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.405244 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.405638 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.405732 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.408267 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:05.408272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:05.408275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:05.408306 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:05.410902 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.411577 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.411581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.411583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.411706 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.412165 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.412168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.412170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.412474 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:05.412679 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.413006 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.413244 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:29:05.420472 28864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:05.427474 28864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:05.466642 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.440655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.27.160.2:39341)
0925 01:29:05.455260 (+ 14605us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.455276 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.455319 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:29:05.455452 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.455464 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.455831 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.457067 (+ 1236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:05.457364 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.459260 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.462852 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.464859 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.465028 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.465315 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.465757 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.465978 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13149,"thread_start_us":13029,"threads_started":1}
I20260925 01:29:05.466868 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.440431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:39341 (local address 127.0.0.1:60610)
0925 01:29:05.454599 (+ 14168us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:05.454665 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:05.454872 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:05.456136 (+ 1264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:05.456148 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:05.456213 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:05.456814 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.456832 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.459442 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.459447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.462644 (+ 3197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.462660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.465314 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.465322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.465502 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.465542 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 01:29:05.465692 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 01:29:05.465931 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13849,"mutex_wait_us":54,"thread_start_us":13678,"threads_started":1}
I20260925 01:29:05.467929 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.440454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:39285 (local address 127.0.0.1:44520)
0925 01:29:05.455332 (+ 14878us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:05.455353 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:05.455508 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:05.458394 (+ 2886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:05.458401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:05.458442 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:05.458850 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.458866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.460747 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.460758 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.463377 (+ 2619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.463392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.466638 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.466646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.466884 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.466925 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 01:29:05.467018 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 01:29:05.467207 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14733,"mutex_wait_us":4,"thread_start_us":14638,"threads_started":1}
I20260925 01:29:05.468436 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.440912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.27.160.3:39285)
0925 01:29:05.457264 (+ 16352us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.457276 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.457323 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:29:05.457855 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.457868 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.458140 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.459051 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:05.459327 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.460610 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.463641 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.465894 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.466078 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.466381 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.467593 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.467820 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16101,"thread_start_us":15900,"threads_started":1}
I20260925 01:29:05.469527 29337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.455691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:37183 (local address 127.0.0.1:48216)
0925 01:29:05.457476 (+ 1785us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:05.457497 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:05.457617 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:05.458941 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:05.458946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:05.458971 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:05.459464 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.459478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.462327 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.462332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.465047 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.465072 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.468461 (+ 3389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.468469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.468718 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.468753 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:29:05.468856 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 01:29:05.468981 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1628,"thread_start_us":1657,"threads_started":1}
I20260925 01:29:05.469738 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.455786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.27.160.1:37183)
0925 01:29:05.458223 (+ 2437us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.458232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.458260 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:29:05.458347 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.458355 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.458614 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.458793 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:05.459850 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.462172 (+ 2322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.465326 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.467528 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.467720 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.468130 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.468921 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.469175 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2257,"thread_start_us":2090,"threads_started":1}
I20260925 01:29:05.473163 29123 tablet_service.cc:1467] Processing CreateTablet for tablet 40ef0b6de1434d71a27e6a072d9ad524 (DEFAULT_TABLE table=test-table [id=d74a1a11d6514e7ab8a0e53b964d6969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:05.473582 29256 tablet_service.cc:1467] Processing CreateTablet for tablet 40ef0b6de1434d71a27e6a072d9ad524 (DEFAULT_TABLE table=test-table [id=d74a1a11d6514e7ab8a0e53b964d6969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:05.474098 28989 tablet_service.cc:1467] Processing CreateTablet for tablet 40ef0b6de1434d71a27e6a072d9ad524 (DEFAULT_TABLE table=test-table [id=d74a1a11d6514e7ab8a0e53b964d6969]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:05.475696 29123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ef0b6de1434d71a27e6a072d9ad524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:05.475716 29256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ef0b6de1434d71a27e6a072d9ad524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:05.475816 28989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ef0b6de1434d71a27e6a072d9ad524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:05.494448 29339 tablet_bootstrap.cc:492] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b: Bootstrap starting.
I20260925 01:29:05.500455 29339 tablet_bootstrap.cc:654] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:05.502290 29340 tablet_bootstrap.cc:492] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53: Bootstrap starting.
I20260925 01:29:05.503448 29339 log.cc:824] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:05.505189 29341 tablet_bootstrap.cc:492] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90: Bootstrap starting.
I20260925 01:29:05.509411 29339 tablet_bootstrap.cc:492] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b: No bootstrap required, opened a new log
I20260925 01:29:05.509814 29340 tablet_bootstrap.cc:654] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:05.509956 29339 ts_tablet_manager.cc:1397] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260925 01:29:05.510469 29341 tablet_bootstrap.cc:654] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:05.512509 29340 log.cc:824] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:05.512785 29341 log.cc:824] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:05.518564 29340 tablet_bootstrap.cc:492] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53: No bootstrap required, opened a new log
I20260925 01:29:05.519021 29341 tablet_bootstrap.cc:492] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90: No bootstrap required, opened a new log
I20260925 01:29:05.519085 29340 ts_tablet_manager.cc:1397] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.013s
I20260925 01:29:05.519616 29341 ts_tablet_manager.cc:1397] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260925 01:29:05.524832 29339 raft_consensus.cc:348] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.525501 29339 raft_consensus.cc:374] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:05.525676 29339 raft_consensus.cc:729] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8409e67458d2436ab7d2520fbd58b53b, State: Initialized, Role: FOLLOWER
I20260925 01:29:05.526624 29339 consensus_queue.cc:260] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.529680 29339 ts_tablet_manager.cc:1428] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b: Time spent starting tablet: real 0.019s user 0.018s sys 0.001s
I20260925 01:29:05.538818 29340 raft_consensus.cc:348] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.539850 29340 raft_consensus.cc:374] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:05.540156 29340 raft_consensus.cc:729] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30606294240f40f8a2d894244b58da53, State: Initialized, Role: FOLLOWER
I20260925 01:29:05.541700 29340 consensus_queue.cc:260] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.541584 29341 raft_consensus.cc:348] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.542462 29341 raft_consensus.cc:374] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:05.542762 29341 raft_consensus.cc:729] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd24bf55a9a147b58a4ed5810d82cc90, State: Initialized, Role: FOLLOWER
I20260925 01:29:05.543996 29321 heartbeater.cc:503] Master 127.27.160.62:42797 was elected leader, sending a full tablet report...
I20260925 01:29:05.544996 29340 ts_tablet_manager.cc:1428] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53: Time spent starting tablet: real 0.026s user 0.014s sys 0.011s
I20260925 01:29:05.545634 29341 consensus_queue.cc:260] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.548462 29341 ts_tablet_manager.cc:1428] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90: Time spent starting tablet: real 0.029s user 0.014s sys 0.009s
I20260925 01:29:05.550072 29188 heartbeater.cc:503] Master 127.27.160.62:42797 was elected leader, sending a full tablet report...
W20260925 01:29:05.594847 29189 tablet.cc:2369] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:29:05.595256 29055 tablet.cc:2369] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:29:05.613415 29322 tablet.cc:2369] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:05.732501 29347 raft_consensus.cc:482] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:05.733103 29347 raft_consensus.cc:504] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.735860 29347 leader_election.cc:288] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30606294240f40f8a2d894244b58da53 (127.27.160.3:39285), 8409e67458d2436ab7d2520fbd58b53b (127.27.160.1:37183)
I20260925 01:29:05.747850 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.736275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:47241 (local address 127.27.160.3:39285)
0925 01:29:05.736594 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.736600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.737999 (+ 1399us) server_negotiation.cc:407] Connection header received
0925 01:29:05.738282 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.738288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.738392 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.738488 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:05.739917 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.741279 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.744647 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.746446 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.746750 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.746868 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.747268 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.747401 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 01:29:05.748152 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.736571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:39285 (local address 127.27.160.2:47241)
0925 01:29:05.737839 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:05.737862 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:05.738080 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:05.738699 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:05.738702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:05.738751 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:05.739725 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.739744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.741445 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.741452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.744364 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.744378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.746827 (+ 2449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.746836 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.747083 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.747110 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 01:29:05.747236 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 01:29:05.747388 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":1,"spinlock_wait_cycles":26880,"thread_start_us":850,"threads_started":1}
I20260925 01:29:05.749471 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.738122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:47383 (local address 127.27.160.1:37183)
0925 01:29:05.738376 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.738382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.739052 (+ 670us) server_negotiation.cc:407] Connection header received
0925 01:29:05.739180 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.739186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.739290 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.739439 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:05.740785 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.742711 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.746315 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.748234 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.748474 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.748602 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.749041 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.749222 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:29:05.749842 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.737820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:37183 (local address 127.27.160.2:47383)
0925 01:29:05.738905 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:05.738919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:05.739055 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:05.739455 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:05.739458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:05.739479 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:05.740540 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.740562 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.742952 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.742956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.745869 (+ 2913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:05.745886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.748617 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.748628 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:05.748823 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.748849 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:29:05.748987 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 01:29:05.749214 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":699,"threads_started":1}
I20260925 01:29:05.749888 29276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ef0b6de1434d71a27e6a072d9ad524" candidate_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30606294240f40f8a2d894244b58da53" is_pre_election: true
I20260925 01:29:05.750749 29276 raft_consensus.cc:2454] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd24bf55a9a147b58a4ed5810d82cc90 in term 0.
I20260925 01:29:05.751545 29009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ef0b6de1434d71a27e6a072d9ad524" candidate_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8409e67458d2436ab7d2520fbd58b53b" is_pre_election: true
I20260925 01:29:05.751884 29079 leader_election.cc:302] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30606294240f40f8a2d894244b58da53, dd24bf55a9a147b58a4ed5810d82cc90; no voters:
I20260925 01:29:05.752383 29009 raft_consensus.cc:2454] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd24bf55a9a147b58a4ed5810d82cc90 in term 0.
I20260925 01:29:05.752619 29347 raft_consensus.cc:2781] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:05.752842 29347 raft_consensus.cc:482] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:05.753010 29347 raft_consensus.cc:3037] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:05.756219 29347 raft_consensus.cc:504] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.756907 29347 leader_election.cc:288] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [CANDIDATE]: Term 1 election: Requested vote from peers 30606294240f40f8a2d894244b58da53 (127.27.160.3:39285), 8409e67458d2436ab7d2520fbd58b53b (127.27.160.1:37183)
I20260925 01:29:05.757622 29276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ef0b6de1434d71a27e6a072d9ad524" candidate_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30606294240f40f8a2d894244b58da53"
I20260925 01:29:05.757702 29009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ef0b6de1434d71a27e6a072d9ad524" candidate_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8409e67458d2436ab7d2520fbd58b53b"
I20260925 01:29:05.757925 29276 raft_consensus.cc:3037] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:05.758031 29009 raft_consensus.cc:3037] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:05.761488 29276 raft_consensus.cc:2454] T 40ef0b6de1434d71a27e6a072d9ad524 P 30606294240f40f8a2d894244b58da53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd24bf55a9a147b58a4ed5810d82cc90 in term 1.
I20260925 01:29:05.762058 29079 leader_election.cc:302] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30606294240f40f8a2d894244b58da53, dd24bf55a9a147b58a4ed5810d82cc90; no voters:
I20260925 01:29:05.762480 29347 raft_consensus.cc:2781] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:05.762470 29009 raft_consensus.cc:2454] T 40ef0b6de1434d71a27e6a072d9ad524 P 8409e67458d2436ab7d2520fbd58b53b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd24bf55a9a147b58a4ed5810d82cc90 in term 1.
I20260925 01:29:05.763420 29347 raft_consensus.cc:686] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [term 1 LEADER]: Becoming Leader. State: Replica: dd24bf55a9a147b58a4ed5810d82cc90, State: Running, Role: LEADER
I20260925 01:29:05.764161 29347 consensus_queue.cc:237] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } }
I20260925 01:29:05.770226 28864 catalog_manager.cc:5495] T 40ef0b6de1434d71a27e6a072d9ad524 P dd24bf55a9a147b58a4ed5810d82cc90 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd24bf55a9a147b58a4ed5810d82cc90 (127.27.160.2). New cstate: current_term: 1 leader_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd24bf55a9a147b58a4ed5810d82cc90" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30606294240f40f8a2d894244b58da53" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 39285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8409e67458d2436ab7d2520fbd58b53b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37183 } health_report { overall_health: UNKNOWN } } }
Sep 25 01:29:05 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:05.844893 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.822861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.27.160.1:37183)
0925 01:29:05.823197 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.823207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.824143 (+ 936us) server_negotiation.cc:407] Connection header received
0925 01:29:05.824315 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.824319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.824389 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.824481 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:05.827268 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.828264 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.830275 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.830882 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.830988 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.836575 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:05.836591 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:05.836607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:05.836651 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:05.840889 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.841882 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.841892 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.841895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.842198 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.842620 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.842628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.842633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.843280 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:05.843759 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.843989 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.844226 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
Sep 25 01:29:05 dist-test-slave-h6hf krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:29:05 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:05.901517 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.884556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.27.160.1:37183)
0925 01:29:05.884777 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.884786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.885306 (+ 520us) server_negotiation.cc:407] Connection header received
0925 01:29:05.885539 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.885550 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.885623 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.885736 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:05.887376 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.888214 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.889698 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.890193 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.890371 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.894899 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:05.894914 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:05.894919 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:05.894969 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:05.898364 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.899186 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.899192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.899195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.899358 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.899947 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.899952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.899954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.900288 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:05.900559 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.900785 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.901118 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 01:29:05 dist-test-slave-h6hf krb5kdc[28820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:29:05 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:05.951584 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.937034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.27.160.1:37183)
0925 01:29:05.937350 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.937362 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.937717 (+ 355us) server_negotiation.cc:407] Connection header received
0925 01:29:05.937848 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.937853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.937991 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.938103 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:05.939432 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.940218 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.941485 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.941972 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.942085 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.946250 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:05.946256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:05.946259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:05.946307 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:05.948965 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.949828 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.949833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.949835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.949957 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.950338 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.950341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.950343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.950648 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:05.950830 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.951140 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.951326 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
W20260925 01:29:05.953514 28969 tablet_service.cc:2908] Rejecting scan request for tablet 40ef0b6de1434d71a27e6a072d9ad524: Uninitialized: clean time has not yet been initialized
I20260925 01:29:05.956717 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 28926
I20260925 01:29:05.970254 29338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:05.957705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.27.160.1:37183)
0925 01:29:05.957867 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:05.957871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:05.958268 (+ 397us) server_negotiation.cc:407] Connection header received
0925 01:29:05.958435 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:05.958439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:05.958511 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:05.958596 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:05.959937 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.960652 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.961956 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:05.962431 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:05.962539 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:05.964793 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:05.964798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:05.964801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:05.964833 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:05.967539 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.968248 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.968256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.968259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.968390 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:05.968772 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:05.968778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:05.968781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:05.969082 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:05.969404 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:05.969691 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 01:29:05.969912 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:29:05.971098 29049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:06.066617 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 28926
I20260925 01:29:06.085517 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29060
Sep 25 01:29:06 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:29:06.101380 29364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:06.086314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.27.160.2:39341)
0925 01:29:06.086790 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:06.086794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:06.086811 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:29:06.086865 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:06.086868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:06.086926 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:06.087025 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:06.088279 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:06.088988 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:06.090381 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:06.090779 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:06.090927 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:06.095588 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:06.095603 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:06.095619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:06.095658 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:06.098557 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:06.099366 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:06.099373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:06.099375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:06.099513 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:06.099920 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:06.099924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:06.099926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:06.100288 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:06.100508 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:06.100811 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 01:29:06.100987 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
I20260925 01:29:06.102031 29183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:06.193734 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 29060
I20260925 01:29:06.210884 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29193
Sep 25 01:29:06 dist-test-slave-h6hf krb5kdc[28820](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:29:06.225526 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:06.211673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.27.160.3:39285)
0925 01:29:06.211856 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:06.211860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:06.211878 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:29:06.212013 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:06.212016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:06.212112 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:06.212194 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:06.213485 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:06.214230 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:06.215464 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:06.215852 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:06.215944 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:06.219926 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:06.219941 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:06.219957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:06.219991 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:06.223065 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:06.223696 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:06.223704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:06.223706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:06.223836 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:06.224203 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:06.224207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:06.224210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:06.224575 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:06.224801 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:06.225015 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 01:29:06.225263 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:29:06.226248 29316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:06.315764 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 29193
I20260925 01:29:06.332203 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 28833
I20260925 01:29:06.333513 28895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:06.441798 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 28833
2026-09-25T01:29:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4450 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:06 dist-test-slave-h6hf krb5kdc[29370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:06 dist-test-slave-h6hf krb5kdc[29370](info): set up 2 sockets
Sep 25 01:29:06 dist-test-slave-h6hf krb5kdc[29370](info): commencing operation
krb5kdc: starting...
W20260925 01:29:07.606241 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:07 dist-test-slave-h6hf krb5kdc[29370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:07.801515 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:33937
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:33937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:08.129653 29383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:08.130021 29383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:08.130201 29383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:08.137436 29383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:08.137607 29383 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:29:08.137681 29383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:08.137753 29383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:08.137804 29383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:08.137868 29383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:08.148489 29383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:33937
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:08.149847 29383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:08.151160 29383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:08.163281 29388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:08.163203 29391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:08.163348 29389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:08.164196 29383 server_base.cc:1034] running on GCE node
I20260925 01:29:08.165035 29383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:08.166567 29383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:08.167846 29383 hybrid_clock.cc:648] HybridClock initialized: now 1790299748167763 us; error 49 us; skew 500 ppm
I20260925 01:29:08.168342 29383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:08 dist-test-slave-h6hf krb5kdc[29370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299748, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:08.171566 29383 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:08.173384 29383 webserver.cc:460] Webserver started at http://127.27.160.62:33939/ using document root <none> and password file <none>
I20260925 01:29:08.174263 29383 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:08.174404 29383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:08.174741 29383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:08.178308 29383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "3c322196f33843389915b23a72ba9d9a"
format_stamp: "Formatted at 2026-09-25 01:29:08 on dist-test-slave-h6hf"
server_key: "5e93ad2f6777ee5010380be8aa991bf0"
server_key_iv: "d7a1fac184c02b140e988b7a2f98ac19"
server_key_version: "encryptionkey@0"
I20260925 01:29:08.179325 29383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "3c322196f33843389915b23a72ba9d9a"
format_stamp: "Formatted at 2026-09-25 01:29:08 on dist-test-slave-h6hf"
server_key: "5e93ad2f6777ee5010380be8aa991bf0"
server_key_iv: "d7a1fac184c02b140e988b7a2f98ac19"
server_key_version: "encryptionkey@0"
I20260925 01:29:08.184826 29383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:29:08.188755 29398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:08.190356 29383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:08.190603 29383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "3c322196f33843389915b23a72ba9d9a"
format_stamp: "Formatted at 2026-09-25 01:29:08 on dist-test-slave-h6hf"
server_key: "5e93ad2f6777ee5010380be8aa991bf0"
server_key_iv: "d7a1fac184c02b140e988b7a2f98ac19"
server_key_version: "encryptionkey@0"
I20260925 01:29:08.190815 29383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:08.212702 29383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:08.217389 29383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:08.217708 29383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:08.240882 29383 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:33937
I20260925 01:29:08.240890 29459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:33937 every 8 connection(s)
I20260925 01:29:08.242398 29383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:08.246186 29460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:08.250065 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 29383
I20260925 01:29:08.250319 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:08.250981 28288 external_mini_cluster.cc:1428] Setting key 74b987054d5dc47a3a1221c280b331da
I20260925 01:29:08.257419 29460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a: Bootstrap starting.
I20260925 01:29:08.261572 29460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:08.262913 29460 log.cc:824] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:08.266533 29460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a: No bootstrap required, opened a new log
Sep 25 01:29:08 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:08.278108 29463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:08.253017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.27.160.62:33937)
0925 01:29:08.253862 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:08.253873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:08.253917 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:29:08.255794 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:08.255819 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:08.256221 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:08.256546 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:08.258324 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.259657 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.261728 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.262479 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.262621 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:08.267931 (+ 5310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:08.267945 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:08.267968 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:08.268027 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:08.272474 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:08.273520 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:08.273531 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:08.273541 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:08.273809 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:08.274253 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:08.274258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:08.274262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:08.275128 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:08.275468 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:08.275966 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 01:29:08.276498 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":357,"threads_started":1}
I20260925 01:29:08.278553 29460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c322196f33843389915b23a72ba9d9a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33937 } }
I20260925 01:29:08.278991 29460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:08.279127 29460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c322196f33843389915b23a72ba9d9a, State: Initialized, Role: FOLLOWER
I20260925 01:29:08.279872 29460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c322196f33843389915b23a72ba9d9a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33937 } }
I20260925 01:29:08.280226 29460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:08.280381 29460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:08.280583 29460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:08.283213 29460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c322196f33843389915b23a72ba9d9a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33937 } }
I20260925 01:29:08.283746 29460 leader_election.cc:302] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c322196f33843389915b23a72ba9d9a; no voters:
I20260925 01:29:08.284409 29460 leader_election.cc:288] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:08.284575 29465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:08.286553 29460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:08.288480 29465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [term 1 LEADER]: Becoming Leader. State: Replica: 3c322196f33843389915b23a72ba9d9a, State: Running, Role: LEADER
I20260925 01:29:08.289409 29465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c322196f33843389915b23a72ba9d9a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33937 } }
I20260925 01:29:08.293238 29466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c322196f33843389915b23a72ba9d9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c322196f33843389915b23a72ba9d9a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33937 } } }
I20260925 01:29:08.293730 29466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:08.294070 29467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c322196f33843389915b23a72ba9d9a. Latest consensus state: current_term: 1 leader_uuid: "3c322196f33843389915b23a72ba9d9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c322196f33843389915b23a72ba9d9a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33937 } } }
I20260925 01:29:08.294659 29467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a [sys.catalog]: This master's current role is: LEADER
W20260925 01:29:08.300516 29480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:29:08.300658 29480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:29:08.300827 29479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:08.303102 29479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:08.312840 29479 catalog_manager.cc:1347] Generated new cluster ID: b788775971fc4767bfdd2dcb36cac2ce
I20260925 01:29:08.313014 29479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:08.328725 29479 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:08.330833 29479 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:08.346805 29479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a: Generated new TSK 0
I20260925 01:29:08.347743 29479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:29:08.413715 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33937
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:08.772064 29486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:08.772387 29486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:08.772549 29486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:08.780478 29486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:08.780607 29486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:08.780759 29486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:29:08.780886 29486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:08.791249 29486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:08.792599 29486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:08.794149 29486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:08.807583 29494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:08.807693 29492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:08.807894 29491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:08.808876 29486 server_base.cc:1034] running on GCE node
I20260925 01:29:08.809590 29486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:08.810564 29486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:08.811834 29486 hybrid_clock.cc:648] HybridClock initialized: now 1790299748811753 us; error 37 us; skew 500 ppm
I20260925 01:29:08.812263 29486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:08 dist-test-slave-h6hf krb5kdc[29370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299748, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:08.819854 29486 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:29:08.821552 29486 webserver.cc:460] Webserver started at http://127.27.160.1:44331/ using document root <none> and password file <none>
I20260925 01:29:08.822286 29486 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:08.822422 29486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:08.822757 29486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:08.825766 29486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "484d2152b4524cddafa6faf8c16d71de"
format_stamp: "Formatted at 2026-09-25 01:29:08 on dist-test-slave-h6hf"
server_key: "fc1e740597814eb69ca2bbeb87b60e99"
server_key_iv: "dcb368601e5b7ae07131c5e52992a08b"
server_key_version: "encryptionkey@0"
I20260925 01:29:08.826573 29486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "484d2152b4524cddafa6faf8c16d71de"
format_stamp: "Formatted at 2026-09-25 01:29:08 on dist-test-slave-h6hf"
server_key: "fc1e740597814eb69ca2bbeb87b60e99"
server_key_iv: "dcb368601e5b7ae07131c5e52992a08b"
server_key_version: "encryptionkey@0"
I20260925 01:29:08.831837 29486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:29:08.835531 29501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:08.837096 29486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:29:08.837400 29486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "484d2152b4524cddafa6faf8c16d71de"
format_stamp: "Formatted at 2026-09-25 01:29:08 on dist-test-slave-h6hf"
server_key: "fc1e740597814eb69ca2bbeb87b60e99"
server_key_iv: "dcb368601e5b7ae07131c5e52992a08b"
server_key_version: "encryptionkey@0"
I20260925 01:29:08.837649 29486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:08.871022 29486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:08.876457 29486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:08.876828 29486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:08.880198 29486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:08.880328 29486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:08.880635 29486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:08.880777 29486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 25 01:29:08 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299748, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:08.904891 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:08.880861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:36315 (local address 127.27.160.62:33937)
0925 01:29:08.881821 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:08.881826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:08.882725 (+ 899us) server_negotiation.cc:407] Connection header received
0925 01:29:08.885469 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:08.885475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:08.885597 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:08.885720 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:08.888250 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.889670 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.892671 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.893094 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.893289 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:08.897297 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:08.897311 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:08.897315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:08.897349 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:08.900683 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:08.901858 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:08.901866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:08.901872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:08.902046 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:08.902708 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:08.902712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:08.902714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:08.903100 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:08.903251 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:08.904296 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0925 01:29:08.904534 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":732,"threads_started":1}
I20260925 01:29:08.906581 29511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:08.881593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33937 (local address 127.27.160.1:36315)
0925 01:29:08.882537 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:08.882576 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:08.885151 (+ 2575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:08.886037 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:08.886055 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:08.886962 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:08.887875 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:08.887900 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.889874 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.889882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:08.892516 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:08.892537 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.893448 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.893462 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:08.893697 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:08.894525 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:08.894546 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:08.897031 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:08.900895 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:08.900905 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:08.900933 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:08.901631 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:08.902175 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:08.902180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:08.902184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:08.902537 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:08.903315 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:08.903321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:08.903823 (+ 502us) client_negotiation.cc:769] Sending connection context
0925 01:29:08.904196 (+ 373us) client_negotiation.cc:240] Negotiation successful
0925 01:29:08.904778 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":483,"threads_started":1}
I20260925 01:29:08.930308 29510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:08.913872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:56839 (local address 127.27.160.62:33937)
0925 01:29:08.914069 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:08.914077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:08.914783 (+ 706us) server_negotiation.cc:407] Connection header received
0925 01:29:08.914944 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:08.914954 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:08.915062 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:08.915197 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:08.916140 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.917573 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.919595 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.920012 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.920156 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:08.923187 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:08.923197 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:08.923203 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:08.923248 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:08.927071 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:08.927892 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:08.927900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:08.927904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:08.928062 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:08.928500 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:08.928507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:08.928511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:08.928824 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:08.929123 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:08.929616 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 01:29:08.929873 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:29:08.930568 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:08.914005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33937 (local address 127.27.160.1:56839)
0925 01:29:08.914662 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:08.914689 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:08.914811 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:08.915215 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:08.915218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:08.915648 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:08.915991 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:08.916001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.917748 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.917760 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:08.919408 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:08.919421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:08.920206 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:08.920213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:08.920445 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:08.921347 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:08.921376 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:08.922877 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:08.927224 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:08.927229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:08.927232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:08.927720 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:08.928196 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:08.928202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:08.928206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:08.928353 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:08.929057 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:08.929063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:08.929309 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 01:29:08.929517 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 01:29:08.929919 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":368,"threads_started":1}
I20260925 01:29:08.940014 29486 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:46359
I20260925 01:29:08.940145 29621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:46359 every 8 connection(s)
I20260925 01:29:08.941414 29486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:08.943868 29622 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33937
I20260925 01:29:08.944166 29622 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:08.945005 29622 heartbeater.cc:511] Master 127.27.160.62:33937 requested a full tablet report, sending...
I20260925 01:29:08.947669 29415 ts_manager.cc:194] Registered new tserver with Master: 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359)
I20260925 01:29:08.948065 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 29486
I20260925 01:29:08.948311 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:08.949045 28288 external_mini_cluster.cc:1428] Setting key d6345e2fbdab649cb68891c1ad9c24b3
I20260925 01:29:08.949349 29415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:36315
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:08.992455 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33937
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:09.348179 29626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:09.348485 29626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:09.348650 29626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:09.356487 29626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:09.356613 29626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:09.356770 29626 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:29:09.356899 29626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:09.366950 29626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:09.368315 29626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:09.369809 29626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:09.384557 29634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:09.384600 29632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:09.384573 29631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:09.391345 29626 server_base.cc:1034] running on GCE node
I20260925 01:29:09.392259 29626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:09.393497 29626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:09.394786 29626 hybrid_clock.cc:648] HybridClock initialized: now 1790299749394730 us; error 61 us; skew 500 ppm
I20260925 01:29:09.395203 29626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:09 dist-test-slave-h6hf krb5kdc[29370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299749, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:09.399119 29626 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:09.400835 29626 webserver.cc:460] Webserver started at http://127.27.160.2:35393/ using document root <none> and password file <none>
I20260925 01:29:09.401705 29626 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:09.401930 29626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:09.402267 29626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:09.405874 29626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "69af43f9aba046e393b5408cc90fa3d3"
format_stamp: "Formatted at 2026-09-25 01:29:09 on dist-test-slave-h6hf"
server_key: "c3ca83876a87e5b4ab8309d037497f7b"
server_key_iv: "6b9f06196b0ac8b0e98e9527921451e5"
server_key_version: "encryptionkey@0"
I20260925 01:29:09.406764 29626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "69af43f9aba046e393b5408cc90fa3d3"
format_stamp: "Formatted at 2026-09-25 01:29:09 on dist-test-slave-h6hf"
server_key: "c3ca83876a87e5b4ab8309d037497f7b"
server_key_iv: "6b9f06196b0ac8b0e98e9527921451e5"
server_key_version: "encryptionkey@0"
I20260925 01:29:09.412950 29626 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 01:29:09.416514 29641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:09.417996 29626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:09.418190 29626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "69af43f9aba046e393b5408cc90fa3d3"
format_stamp: "Formatted at 2026-09-25 01:29:09 on dist-test-slave-h6hf"
server_key: "c3ca83876a87e5b4ab8309d037497f7b"
server_key_iv: "6b9f06196b0ac8b0e98e9527921451e5"
server_key_version: "encryptionkey@0"
I20260925 01:29:09.418440 29626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:09.437809 29626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:09.442235 29626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:09.442510 29626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:09.445446 29626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:09.445565 29626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:09.445729 29626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:09.445803 29626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:29:09 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299749, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:09.471086 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:09.446409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:41447 (local address 127.27.160.62:33937)
0925 01:29:09.446869 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:09.446875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:09.448155 (+ 1280us) server_negotiation.cc:407] Connection header received
0925 01:29:09.450955 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:09.450961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:09.451064 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:09.451187 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:09.453548 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:09.454953 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:09.456726 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:09.457540 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:09.457679 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:09.463196 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:09.463204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:09.463208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:09.463246 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:09.466432 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:09.467617 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:09.467622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:09.467624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:09.467863 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:09.468744 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:09.468751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:09.468760 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:09.469242 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:09.469440 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:09.470341 (+ 901us) server_negotiation.cc:299] Negotiation successful
0925 01:29:09.470619 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":216,"threads_started":1}
I20260925 01:29:09.472442 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:09.447051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33937 (local address 127.27.160.2:41447)
0925 01:29:09.447970 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:09.448018 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:09.450519 (+ 2501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:09.451453 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:09.451462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:09.452375 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:09.453303 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:09.453330 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:09.455167 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:09.455171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:09.456555 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:09.456569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:09.457770 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:09.457775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:09.458092 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:09.459006 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:09.459031 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:09.462882 (+ 3851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:09.466597 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:09.466611 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:09.466636 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:09.467430 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:09.468173 (+ 743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:09.468180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:09.468184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:09.468606 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:09.469412 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:09.469425 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:09.469880 (+ 455us) client_negotiation.cc:769] Sending connection context
0925 01:29:09.470231 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 01:29:09.470818 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":392,"threads_started":1}
I20260925 01:29:09.496017 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:09.479616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33937 (local address 127.27.160.2:49363)
0925 01:29:09.480319 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:09.480346 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:09.480547 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:09.480910 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:09.480918 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:09.481764 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:09.482282 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:09.482302 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:09.483625 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:09.483630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:09.485363 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:09.485387 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:09.486276 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:09.486282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:09.486488 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:09.487316 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:09.487335 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:09.488272 (+ 937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:09.492177 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:09.492187 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:09.492194 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:09.493047 (+ 853us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:09.493606 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:09.493611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:09.493616 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:09.493845 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:09.494728 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:09.494735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:09.495057 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 01:29:09.495264 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 01:29:09.495554 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":428,"threads_started":1}
I20260925 01:29:09.496769 29650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:09.479942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:49363 (local address 127.27.160.62:33937)
0925 01:29:09.480150 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:09.480156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:09.480455 (+ 299us) server_negotiation.cc:407] Connection header received
0925 01:29:09.480675 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:09.480679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:09.480755 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:09.480849 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:09.482533 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:09.483455 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:09.485585 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:09.486063 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:09.486207 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:09.488466 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:09.488472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:09.488483 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:09.488518 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:09.491943 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:09.493250 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:09.493260 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:09.493264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:09.493428 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:09.494034 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:09.494041 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:09.494044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:09.494503 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:09.494655 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:09.496374 (+ 1719us) server_negotiation.cc:299] Negotiation successful
0925 01:29:09.496553 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 01:29:09.505714 29626 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:44997
I20260925 01:29:09.505718 29761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:44997 every 8 connection(s)
I20260925 01:29:09.506945 29626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:09.509481 29762 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33937
I20260925 01:29:09.509704 29762 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:09.510497 29762 heartbeater.cc:511] Master 127.27.160.62:33937 requested a full tablet report, sending...
I20260925 01:29:09.511996 29415 ts_manager.cc:194] Registered new tserver with Master: 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2:44997)
I20260925 01:29:09.512838 29415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:41447
I20260925 01:29:09.513343 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 29626
I20260925 01:29:09.513592 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:09.514106 28288 external_mini_cluster.cc:1428] Setting key e9e0a9ad40adcf9e81a923fa1d635551
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:09.557097 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33937
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:09.909539 29766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:09.909868 29766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:09.910048 29766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:09.917501 29766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:09.917640 29766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:09.917794 29766 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:29:09.917934 29766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:09.927917 29766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:09.929297 29766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:09.930693 29766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:09.944837 29771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:09.944744 29772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:09.945190 29774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:09.945883 29766 server_base.cc:1034] running on GCE node
I20260925 01:29:09.946487 29766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:09.947587 29766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:09.948868 29766 hybrid_clock.cc:648] HybridClock initialized: now 1790299749948704 us; error 131 us; skew 500 ppm
I20260925 01:29:09.949391 29766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:09 dist-test-slave-h6hf krb5kdc[29370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299749, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:09.953054 29622 heartbeater.cc:503] Master 127.27.160.62:33937 was elected leader, sending a full tablet report...
I20260925 01:29:09.953473 29766 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:09.955304 29766 webserver.cc:460] Webserver started at http://127.27.160.3:44243/ using document root <none> and password file <none>
I20260925 01:29:09.956039 29766 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:09.956166 29766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:09.956491 29766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:09.959548 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "723d26a819f54a4d810288ae60733cd6"
format_stamp: "Formatted at 2026-09-25 01:29:09 on dist-test-slave-h6hf"
server_key: "cb8f14def3d7f88e12de55f829a74ada"
server_key_iv: "3553913bae0208c0390c861563b1dcb5"
server_key_version: "encryptionkey@0"
I20260925 01:29:09.960358 29766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "723d26a819f54a4d810288ae60733cd6"
format_stamp: "Formatted at 2026-09-25 01:29:09 on dist-test-slave-h6hf"
server_key: "cb8f14def3d7f88e12de55f829a74ada"
server_key_iv: "3553913bae0208c0390c861563b1dcb5"
server_key_version: "encryptionkey@0"
I20260925 01:29:09.965708 29766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:29:09.969501 29781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:09.971032 29766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:09.971220 29766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "723d26a819f54a4d810288ae60733cd6"
format_stamp: "Formatted at 2026-09-25 01:29:09 on dist-test-slave-h6hf"
server_key: "cb8f14def3d7f88e12de55f829a74ada"
server_key_iv: "3553913bae0208c0390c861563b1dcb5"
server_key_version: "encryptionkey@0"
I20260925 01:29:09.971446 29766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:09.995316 29766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:09.999936 29766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:10.000231 29766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:10.003067 29766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:10.003175 29766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:10.003372 29766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:10.003468 29766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:29:10 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299749, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:10.029049 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.004289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:33217 (local address 127.27.160.62:33937)
0925 01:29:10.009535 (+ 5246us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.009540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.009562 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:29:10.009620 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.009625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.009691 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.009829 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:10.011747 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.012583 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.015487 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.015974 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.016127 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.020957 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:10.020968 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:10.020977 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:10.021034 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:10.025392 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:10.026414 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:10.026420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:10.026425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:10.026571 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:10.027066 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:10.027072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:10.027075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:10.027380 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:10.027560 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.028355 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.028614 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5084,"thread_start_us":4982,"threads_started":1}
I20260925 01:29:10.030332 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.004928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33937 (local address 127.27.160.3:33217)
0925 01:29:10.006045 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.006103 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.008172 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.010168 (+ 1996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.010183 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.010887 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:10.011538 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.011558 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.012755 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.012762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.015200 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.015252 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.016247 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.016255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.016508 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.017379 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:10.017424 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:10.020622 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:10.025624 (+ 5002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:10.025636 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:10.025674 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:10.026256 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:10.026695 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:10.026700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:10.026704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:10.026929 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:10.027535 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:10.027544 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:10.027949 (+ 405us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.028241 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.028817 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":542,"threads_started":1}
I20260925 01:29:10.057044 29845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.040214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33937 (local address 127.27.160.3:60737)
0925 01:29:10.040839 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.040861 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.041053 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.041778 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.041781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.042594 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:10.043057 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.043068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.044178 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.044184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.045449 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.045474 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.046769 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.046779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.047090 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.047966 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:10.047987 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:10.049485 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:10.054188 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:10.054197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:10.054202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:10.054661 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:10.055085 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:10.055092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:10.055096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:10.055241 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:10.056039 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:10.056045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:10.056298 (+ 253us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.056419 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.056677 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":355,"threads_started":1}
I20260925 01:29:10.057993 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.040019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:60737 (local address 127.27.160.62:33937)
0925 01:29:10.040199 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.040204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.040970 (+ 766us) server_negotiation.cc:407] Connection header received
0925 01:29:10.041311 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.041321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.041442 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.041537 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:10.043224 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.044024 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.045668 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.046466 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.046642 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.049711 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:10.049720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:10.049729 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:10.049772 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:10.053961 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:10.054814 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:10.054820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:10.054824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:10.054965 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:10.055382 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:10.055393 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:10.055397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:10.055856 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:10.057431 (+ 1575us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.057551 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.057753 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:29:10.067736 29766 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:44025
I20260925 01:29:10.067775 29901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:44025 every 8 connection(s)
I20260925 01:29:10.069006 29766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:10.071118 29902 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33937
I20260925 01:29:10.071368 29902 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:10.072003 29902 heartbeater.cc:511] Master 127.27.160.62:33937 requested a full tablet report, sending...
I20260925 01:29:10.073416 29415 ts_manager.cc:194] Registered new tserver with Master: 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025)
I20260925 01:29:10.074273 29415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:33217
I20260925 01:29:10.077783 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 29766
I20260925 01:29:10.078071 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:10.078606 28288 external_mini_cluster.cc:1428] Setting key e1a53ef4d9fdd2a438f47fd2038d60f0
I20260925 01:29:10.082180 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:29:10 dist-test-slave-h6hf krb5kdc[29370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:29:10 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:10.132510 29790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.116549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.27.160.62:33937)
0925 01:29:10.116840 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.116850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.117169 (+ 319us) server_negotiation.cc:407] Connection header received
0925 01:29:10.117444 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.117450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.117522 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.117644 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:10.119211 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.120033 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.121692 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.122148 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.122310 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.126650 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:10.126656 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:10.126660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:10.126699 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:10.129546 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:10.130243 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:10.130248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:10.130250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:10.130425 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:10.130886 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:10.130891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:10.130893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:10.131205 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:10.131505 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.131851 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.132098 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 01:29:10.137050 29415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:10.144719 29415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:10.176124 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.160886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:44025 (local address 127.0.0.1:48722)
0925 01:29:10.165718 (+ 4832us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.165743 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.166004 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.167105 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.167113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.167167 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:10.167738 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.167750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.169535 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.169542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.172499 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.172509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.175162 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.175167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.175382 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.175421 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.175523 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.175667 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4656,"mutex_wait_us":131,"spinlock_wait_cycles":27136,"thread_start_us":4481,"threads_started":1}
I20260925 01:29:10.176614 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.160009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:44997 (local address 127.0.0.1:52500)
0925 01:29:10.160891 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.160943 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.161198 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.166992 (+ 5794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.167001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.167035 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:10.167555 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.167568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.169796 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.169804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.172200 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.172215 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.174906 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.174916 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.175163 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.175237 (+ 74us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.175375 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.175624 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":33,"thread_start_us":383,"threads_started":1}
I20260925 01:29:10.176869 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.160958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.27.160.3:44025)
0925 01:29:10.162393 (+ 1435us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.162407 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.165876 (+ 3469us) server_negotiation.cc:407] Connection header received
0925 01:29:10.166223 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.166233 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.166550 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.166783 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:10.167965 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.169397 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.172655 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.174551 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.174701 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.175010 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.176391 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.176562 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224,"thread_start_us":1041,"threads_started":1}
I20260925 01:29:10.176932 29920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.161912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.27.160.1:46359)
0925 01:29:10.166543 (+ 4631us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.166552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.167304 (+ 752us) server_negotiation.cc:407] Connection header received
0925 01:29:10.167414 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.167423 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.167685 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.167885 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:10.168751 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.170328 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.173055 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.174767 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.175013 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.175322 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.176398 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.176606 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4471,"thread_start_us":4334,"threads_started":1}
I20260925 01:29:10.177474 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.161370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46359 (local address 127.0.0.1:57672)
0925 01:29:10.167008 (+ 5638us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.167027 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.167166 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.168149 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.168154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.168182 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:10.168537 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.168551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.170528 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.170533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.172831 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.172846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.175479 (+ 2633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.175486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.175667 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.175698 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.175793 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.175963 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5474,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":5327,"threads_started":1}
I20260925 01:29:10.177729 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.160245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52500 (local address 127.27.160.2:44997)
0925 01:29:10.166200 (+ 5955us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.166215 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.166256 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:29:10.166374 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.166383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.166695 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.167816 (+ 1121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:10.168039 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.169644 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.172388 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.174114 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.176856 (+ 2742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.177085 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.177205 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.177385 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5727,"thread_start_us":5581,"threads_started":1}
I20260925 01:29:10.180991 29692 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbe07c5f0294b76bda1556354362fd8 (DEFAULT_TABLE table=test-table [id=4e6f789f669b4e63b84f8ee0a2b788a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:10.181003 29552 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbe07c5f0294b76bda1556354362fd8 (DEFAULT_TABLE table=test-table [id=4e6f789f669b4e63b84f8ee0a2b788a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:10.181003 29836 tablet_service.cc:1467] Processing CreateTablet for tablet 9cbe07c5f0294b76bda1556354362fd8 (DEFAULT_TABLE table=test-table [id=4e6f789f669b4e63b84f8ee0a2b788a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:10.182447 29692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbe07c5f0294b76bda1556354362fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.182531 29836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbe07c5f0294b76bda1556354362fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.182530 29552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cbe07c5f0294b76bda1556354362fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.196326 29921 tablet_bootstrap.cc:492] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Bootstrap starting.
I20260925 01:29:10.200709 29921 tablet_bootstrap.cc:654] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.202121 29921 log.cc:824] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:10.204730 29923 tablet_bootstrap.cc:492] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Bootstrap starting.
I20260925 01:29:10.205762 29922 tablet_bootstrap.cc:492] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Bootstrap starting.
I20260925 01:29:10.205972 29921 tablet_bootstrap.cc:492] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: No bootstrap required, opened a new log
I20260925 01:29:10.206305 29921 ts_tablet_manager.cc:1397] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 01:29:10.210450 29923 tablet_bootstrap.cc:654] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.210953 29922 tablet_bootstrap.cc:654] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.212695 29922 log.cc:824] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:10.213374 29923 log.cc:824] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:10.217692 29922 tablet_bootstrap.cc:492] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: No bootstrap required, opened a new log
I20260925 01:29:10.218084 29922 ts_tablet_manager.cc:1397] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260925 01:29:10.220587 29923 tablet_bootstrap.cc:492] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: No bootstrap required, opened a new log
I20260925 01:29:10.221189 29923 ts_tablet_manager.cc:1397] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.004s
I20260925 01:29:10.224036 29921 raft_consensus.cc:348] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.224915 29921 raft_consensus.cc:374] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:10.225108 29921 raft_consensus.cc:729] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69af43f9aba046e393b5408cc90fa3d3, State: Initialized, Role: FOLLOWER
I20260925 01:29:10.226307 29921 consensus_queue.cc:260] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.230196 29922 raft_consensus.cc:348] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.230605 29922 raft_consensus.cc:374] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:10.230768 29922 raft_consensus.cc:729] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723d26a819f54a4d810288ae60733cd6, State: Initialized, Role: FOLLOWER
I20260925 01:29:10.231644 29922 consensus_queue.cc:260] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.231900 29762 heartbeater.cc:503] Master 127.27.160.62:33937 was elected leader, sending a full tablet report...
I20260925 01:29:10.233140 29902 heartbeater.cc:503] Master 127.27.160.62:33937 was elected leader, sending a full tablet report...
I20260925 01:29:10.234022 29922 ts_tablet_manager.cc:1428] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260925 01:29:10.236675 29921 ts_tablet_manager.cc:1428] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Time spent starting tablet: real 0.030s user 0.014s sys 0.012s
I20260925 01:29:10.238137 29923 raft_consensus.cc:348] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.238736 29923 raft_consensus.cc:374] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:10.238866 29923 raft_consensus.cc:729] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484d2152b4524cddafa6faf8c16d71de, State: Initialized, Role: FOLLOWER
I20260925 01:29:10.239585 29923 consensus_queue.cc:260] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.241765 29923 ts_tablet_manager.cc:1428] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
W20260925 01:29:10.259176 29763 tablet.cc:2369] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:29:10.320672 29903 tablet.cc:2369] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:29:10.444268 29623 tablet.cc:2369] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:10.479207 29927 raft_consensus.cc:482] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:10.479732 29928 raft_consensus.cc:482] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:10.479730 29927 raft_consensus.cc:504] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.480018 29928 raft_consensus.cc:504] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.481650 29928 leader_election.cc:288] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2:44997), 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359)
I20260925 01:29:10.481769 29927 leader_election.cc:288] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359), 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025)
I20260925 01:29:10.491385 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.482399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:44997 (local address 127.27.160.3:36259)
0925 01:29:10.482718 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.482738 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.482939 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.483920 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.483932 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.483962 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:10.484205 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.484220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.486013 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.486019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.488320 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.488338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.490377 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.490384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.490640 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.490679 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.490784 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.491032 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260925 01:29:10.492416 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.483254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:36259 (local address 127.27.160.2:44997)
0925 01:29:10.483446 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.483455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.483471 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:29:10.483536 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.483542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.483617 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.483709 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:10.484399 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.485840 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.488562 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.490023 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.490192 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.490275 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.491974 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.492125 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":70}
I20260925 01:29:10.492585 29920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.482303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:39753 (local address 127.27.160.1:46359)
0925 01:29:10.482595 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.482605 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.483421 (+ 816us) server_negotiation.cc:407] Connection header received
0925 01:29:10.483744 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.483749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.483836 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.483908 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:10.485493 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.486677 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.489779 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.491264 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.491407 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.491501 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.492189 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.492331 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 01:29:10.493588 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.482044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46359 (local address 127.27.160.2:39753)
0925 01:29:10.483299 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.483320 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.483537 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.484116 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.484123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.484159 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:10.485311 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.485326 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.486910 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.486917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.489527 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.489544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.491768 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.491775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.492003 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.492035 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.492917 (+ 882us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.493106 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":2,"thread_start_us":870,"threads_started":1}
I20260925 01:29:10.494745 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbe07c5f0294b76bda1556354362fd8" candidate_uuid: "723d26a819f54a4d810288ae60733cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69af43f9aba046e393b5408cc90fa3d3" is_pre_election: true
I20260925 01:29:10.494751 29577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbe07c5f0294b76bda1556354362fd8" candidate_uuid: "69af43f9aba046e393b5408cc90fa3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484d2152b4524cddafa6faf8c16d71de" is_pre_election: true
I20260925 01:29:10.495287 29717 raft_consensus.cc:2454] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723d26a819f54a4d810288ae60733cd6 in term 0.
I20260925 01:29:10.495389 29577 raft_consensus.cc:2454] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69af43f9aba046e393b5408cc90fa3d3 in term 0.
I20260925 01:29:10.496102 29644 leader_election.cc:302] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d2152b4524cddafa6faf8c16d71de, 69af43f9aba046e393b5408cc90fa3d3; no voters:
I20260925 01:29:10.496690 29927 raft_consensus.cc:2781] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:10.496668 29784 leader_election.cc:302] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69af43f9aba046e393b5408cc90fa3d3, 723d26a819f54a4d810288ae60733cd6; no voters:
I20260925 01:29:10.496882 29927 raft_consensus.cc:482] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:10.497056 29927 raft_consensus.cc:3037] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:10.497290 29928 raft_consensus.cc:2781] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:10.497434 29928 raft_consensus.cc:482] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:10.497521 29928 raft_consensus.cc:3037] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:10.500900 29927 raft_consensus.cc:504] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.501351 29928 raft_consensus.cc:504] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.501910 29918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.483261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:51961 (local address 127.27.160.3:44025)
0925 01:29:10.483421 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.483432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.493606 (+ 10174us) server_negotiation.cc:407] Connection header received
0925 01:29:10.493882 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.493888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.493978 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.494039 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:10.494763 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.495845 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.498832 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.500637 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.500826 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.500906 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.501381 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.501543 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":58}
I20260925 01:29:10.502203 29577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbe07c5f0294b76bda1556354362fd8" candidate_uuid: "69af43f9aba046e393b5408cc90fa3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484d2152b4524cddafa6faf8c16d71de"
I20260925 01:29:10.502279 29928 leader_election.cc:288] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [CANDIDATE]: Term 1 election: Requested vote from peers 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2:44997), 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359)
I20260925 01:29:10.502419 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.482820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46359 (local address 127.27.160.3:39919)
0925 01:29:10.491679 (+ 8859us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.491701 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.491823 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.493189 (+ 1366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.493196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.493229 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:10.493493 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.493509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.495964 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.495970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.497708 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.497726 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.499757 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.499765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.501067 (+ 1302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.501113 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.501240 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.501421 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8736,"mutex_wait_us":4,"thread_start_us":10612,"threads_started":1}
I20260925 01:29:10.502424 29577 raft_consensus.cc:3037] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:10.502763 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.483368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:39919 (local address 127.27.160.1:46359)
0925 01:29:10.484952 (+ 1584us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.484958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.492613 (+ 7655us) server_negotiation.cc:407] Connection header received
0925 01:29:10.492683 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.492689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.492772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.492880 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:10.493749 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.495740 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.497930 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.499212 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.499392 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.499583 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.501701 (+ 2118us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.501862 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1435,"thread_start_us":1165,"threads_started":1}
I20260925 01:29:10.502920 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.482242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:44025 (local address 127.27.160.2:51961)
0925 01:29:10.493502 (+ 11260us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.493525 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.493743 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.494221 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.494226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.494254 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:10.494565 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.494573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.496100 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.496105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.498608 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.498624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.500920 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.500928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.501103 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.501214 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.502159 (+ 945us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.502321 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11112,"thread_start_us":10880,"threads_started":1}
I20260925 01:29:10.503234 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbe07c5f0294b76bda1556354362fd8" candidate_uuid: "723d26a819f54a4d810288ae60733cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69af43f9aba046e393b5408cc90fa3d3"
I20260925 01:29:10.503526 29927 leader_election.cc:288] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [CANDIDATE]: Term 1 election: Requested vote from peers 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359), 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025)
I20260925 01:29:10.503590 29717 raft_consensus.cc:2379] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 723d26a819f54a4d810288ae60733cd6 in current term 1: Already voted for candidate 69af43f9aba046e393b5408cc90fa3d3 in this term.
I20260925 01:29:10.503856 29576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbe07c5f0294b76bda1556354362fd8" candidate_uuid: "723d26a819f54a4d810288ae60733cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484d2152b4524cddafa6faf8c16d71de" is_pre_election: true
I20260925 01:29:10.504524 29856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbe07c5f0294b76bda1556354362fd8" candidate_uuid: "69af43f9aba046e393b5408cc90fa3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723d26a819f54a4d810288ae60733cd6"
I20260925 01:29:10.505097 29856 raft_consensus.cc:2379] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 69af43f9aba046e393b5408cc90fa3d3 in current term 1: Already voted for candidate 723d26a819f54a4d810288ae60733cd6 in this term.
I20260925 01:29:10.505733 29857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbe07c5f0294b76bda1556354362fd8" candidate_uuid: "69af43f9aba046e393b5408cc90fa3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723d26a819f54a4d810288ae60733cd6" is_pre_election: true
I20260925 01:29:10.506207 29857 raft_consensus.cc:2379] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 69af43f9aba046e393b5408cc90fa3d3 in current term 1: Already voted for candidate 723d26a819f54a4d810288ae60733cd6 in this term.
I20260925 01:29:10.506760 29577 raft_consensus.cc:2454] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69af43f9aba046e393b5408cc90fa3d3 in term 1.
I20260925 01:29:10.507162 29576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cbe07c5f0294b76bda1556354362fd8" candidate_uuid: "723d26a819f54a4d810288ae60733cd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484d2152b4524cddafa6faf8c16d71de"
I20260925 01:29:10.507320 29644 leader_election.cc:302] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d2152b4524cddafa6faf8c16d71de, 69af43f9aba046e393b5408cc90fa3d3; no voters:
I20260925 01:29:10.507593 29576 raft_consensus.cc:2379] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 723d26a819f54a4d810288ae60733cd6 in current term 1: Already voted for candidate 69af43f9aba046e393b5408cc90fa3d3 in this term.
I20260925 01:29:10.507920 29927 raft_consensus.cc:2781] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:10.508225 29784 leader_election.cc:302] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 723d26a819f54a4d810288ae60733cd6; no voters: 484d2152b4524cddafa6faf8c16d71de, 69af43f9aba046e393b5408cc90fa3d3
I20260925 01:29:10.508904 29928 raft_consensus.cc:2726] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 01:29:10.509645 29927 raft_consensus.cc:686] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 LEADER]: Becoming Leader. State: Replica: 69af43f9aba046e393b5408cc90fa3d3, State: Running, Role: LEADER
I20260925 01:29:10.510504 29927 consensus_queue.cc:237] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.517107 29414 catalog_manager.cc:5495] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2). New cstate: current_term: 1 leader_uuid: "69af43f9aba046e393b5408cc90fa3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } health_report { overall_health: UNKNOWN } } }
I20260925 01:29:10.572736 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.566607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.27.160.2:44997)
0925 01:29:10.566805 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.566811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.567200 (+ 389us) server_negotiation.cc:407] Connection header received
0925 01:29:10.567555 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.567561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.567657 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.567812 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:10.568204 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.569027 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.570429 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.570854 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.571014 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.571535 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:10.572085 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:10.572253 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.572344 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.572431 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 01:29:10.582875 29857 raft_consensus.cc:1261] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Refusing update from remote peer 69af43f9aba046e393b5408cc90fa3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:10.583036 29576 raft_consensus.cc:1261] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Refusing update from remote peer 69af43f9aba046e393b5408cc90fa3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:10.584097 29934 consensus_queue.cc:1035] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:10.584517 29927 consensus_queue.cc:1035] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:10.596508 29937 mvcc.cc:205] Tried to move back new op lower bound from 7333067778374901760 to 7333067778096029696. Current Snapshot: MvccSnapshot[applied={T|T < 7333067778374901760}]
I20260925 01:29:10.597278 29938 mvcc.cc:205] Tried to move back new op lower bound from 7333067778374901760 to 7333067778096029696. Current Snapshot: MvccSnapshot[applied={T|T < 7333067778374901760}]
I20260925 01:29:10.602094 29939 mvcc.cc:205] Tried to move back new op lower bound from 7333067778374901760 to 7333067778096029696. Current Snapshot: MvccSnapshot[applied={T|T < 7333067778374901760}]
I20260925 01:29:10.617789 29415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:10.618191 29415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:10.622457 29415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a: Sending DeleteTablet for 3 replicas of tablet 9cbe07c5f0294b76bda1556354362fd8
I20260925 01:29:10.624058 29692 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cbe07c5f0294b76bda1556354362fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:10 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:52500
I20260925 01:29:10.624079 29836 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cbe07c5f0294b76bda1556354362fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:10 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:48722
I20260925 01:29:10.624879 29552 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cbe07c5f0294b76bda1556354362fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:10 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:57672
I20260925 01:29:10.633957 29415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:10.634904 29415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:10.639910 29948 tablet_replica.cc:330] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: stopping tablet replica
I20260925 01:29:10.640081 29946 tablet_replica.cc:330] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: stopping tablet replica
I20260925 01:29:10.640695 29946 raft_consensus.cc:2229] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:10.640802 29948 raft_consensus.cc:2229] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:10.641930 29948 raft_consensus.cc:2258] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:10.642009 29946 raft_consensus.cc:2258] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:10.644459 29948 ts_tablet_manager.cc:1905] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:10.644479 29946 ts_tablet_manager.cc:1905] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:10.645947 29834 tablet_service.cc:1467] Processing CreateTablet for tablet 0f77dc287584455d96ee5447950d4212 (DEFAULT_TABLE table=test-table [id=b60b82baacd146dc8d0f993a3aa93fe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:10.646049 29692 tablet_service.cc:1467] Processing CreateTablet for tablet 0f77dc287584455d96ee5447950d4212 (DEFAULT_TABLE table=test-table [id=b60b82baacd146dc8d0f993a3aa93fe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:10.646739 29692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f77dc287584455d96ee5447950d4212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.646894 29834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f77dc287584455d96ee5447950d4212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.648030 29947 tablet_replica.cc:330] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: stopping tablet replica
I20260925 01:29:10.648756 29947 raft_consensus.cc:2229] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:10.649659 29947 raft_consensus.cc:2258] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:10.651258 29552 tablet_service.cc:1467] Processing CreateTablet for tablet 0f77dc287584455d96ee5447950d4212 (DEFAULT_TABLE table=test-table [id=b60b82baacd146dc8d0f993a3aa93fe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:10.651873 29552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f77dc287584455d96ee5447950d4212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.652328 29947 ts_tablet_manager.cc:1905] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:10.660064 29922 tablet_bootstrap.cc:492] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: Bootstrap starting.
I20260925 01:29:10.663228 29922 tablet_bootstrap.cc:654] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.665728 29921 tablet_bootstrap.cc:492] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: Bootstrap starting.
I20260925 01:29:10.666867 29948 ts_tablet_manager.cc:1918] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:10.667163 29948 log.cc:1196] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/9cbe07c5f0294b76bda1556354362fd8
I20260925 01:29:10.668114 29948 ts_tablet_manager.cc:1939] T 9cbe07c5f0294b76bda1556354362fd8 P 484d2152b4524cddafa6faf8c16d71de: Deleting consensus metadata
I20260925 01:29:10.668112 29947 ts_tablet_manager.cc:1918] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:10.668375 29947 log.cc:1196] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/9cbe07c5f0294b76bda1556354362fd8
I20260925 01:29:10.669256 29947 ts_tablet_manager.cc:1939] T 9cbe07c5f0294b76bda1556354362fd8 P 723d26a819f54a4d810288ae60733cd6: Deleting consensus metadata
I20260925 01:29:10.670450 29921 tablet_bootstrap.cc:654] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.671986 29402 catalog_manager.cc:4841] TS 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025): tablet 9cbe07c5f0294b76bda1556354362fd8 (table test-table [id=4e6f789f669b4e63b84f8ee0a2b788a1]) successfully deleted
I20260925 01:29:10.675916 29922 tablet_bootstrap.cc:492] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: No bootstrap required, opened a new log
I20260925 01:29:10.676182 29922 ts_tablet_manager.cc:1397] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20260925 01:29:10.676610 29921 tablet_bootstrap.cc:492] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: No bootstrap required, opened a new log
I20260925 01:29:10.676836 29921 ts_tablet_manager.cc:1397] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20260925 01:29:10.677556 29922 raft_consensus.cc:348] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.677875 29922 raft_consensus.cc:374] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:10.678038 29946 ts_tablet_manager.cc:1918] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:10.678089 29922 raft_consensus.cc:729] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723d26a819f54a4d810288ae60733cd6, State: Initialized, Role: FOLLOWER
I20260925 01:29:10.678319 29946 log.cc:1196] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/9cbe07c5f0294b76bda1556354362fd8
I20260925 01:29:10.678597 29922 consensus_queue.cc:260] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.678966 29401 catalog_manager.cc:4841] TS 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359): tablet 9cbe07c5f0294b76bda1556354362fd8 (table test-table [id=4e6f789f669b4e63b84f8ee0a2b788a1]) successfully deleted
I20260925 01:29:10.679217 29921 raft_consensus.cc:348] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.679591 29921 raft_consensus.cc:374] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:10.679644 29922 ts_tablet_manager.cc:1428] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20260925 01:29:10.679942 29921 raft_consensus.cc:729] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69af43f9aba046e393b5408cc90fa3d3, State: Initialized, Role: FOLLOWER
I20260925 01:29:10.680429 29921 consensus_queue.cc:260] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.680656 29923 tablet_bootstrap.cc:492] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: Bootstrap starting.
I20260925 01:29:10.681529 29921 ts_tablet_manager.cc:1428] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:29:10.683672 29923 tablet_bootstrap.cc:654] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.686200 29946 ts_tablet_manager.cc:1939] T 9cbe07c5f0294b76bda1556354362fd8 P 69af43f9aba046e393b5408cc90fa3d3: Deleting consensus metadata
I20260925 01:29:10.687664 29923 tablet_bootstrap.cc:492] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: No bootstrap required, opened a new log
I20260925 01:29:10.687887 29923 ts_tablet_manager.cc:1397] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260925 01:29:10.688630 29401 catalog_manager.cc:4841] TS 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2:44997): tablet 9cbe07c5f0294b76bda1556354362fd8 (table test-table [id=4e6f789f669b4e63b84f8ee0a2b788a1]) successfully deleted
I20260925 01:29:10.689081 29923 raft_consensus.cc:348] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.689424 29923 raft_consensus.cc:374] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:10.689618 29923 raft_consensus.cc:729] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484d2152b4524cddafa6faf8c16d71de, State: Initialized, Role: FOLLOWER
I20260925 01:29:10.690053 29923 consensus_queue.cc:260] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.690846 29923 ts_tablet_manager.cc:1428] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:10.857636 29934 raft_consensus.cc:482] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:10.858081 29934 raft_consensus.cc:504] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.859094 29934 leader_election.cc:288] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359), 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025)
I20260925 01:29:10.859822 29576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f77dc287584455d96ee5447950d4212" candidate_uuid: "69af43f9aba046e393b5408cc90fa3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484d2152b4524cddafa6faf8c16d71de" is_pre_election: true
I20260925 01:29:10.859797 29857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f77dc287584455d96ee5447950d4212" candidate_uuid: "69af43f9aba046e393b5408cc90fa3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723d26a819f54a4d810288ae60733cd6" is_pre_election: true
I20260925 01:29:10.860160 29576 raft_consensus.cc:2454] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69af43f9aba046e393b5408cc90fa3d3 in term 0.
I20260925 01:29:10.860183 29857 raft_consensus.cc:2454] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69af43f9aba046e393b5408cc90fa3d3 in term 0.
I20260925 01:29:10.860716 29644 leader_election.cc:302] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d2152b4524cddafa6faf8c16d71de, 69af43f9aba046e393b5408cc90fa3d3; no voters:
I20260925 01:29:10.861066 29934 raft_consensus.cc:2781] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:10.861230 29934 raft_consensus.cc:482] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:10.861342 29934 raft_consensus.cc:3037] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:10.864331 29934 raft_consensus.cc:504] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.864969 29934 leader_election.cc:288] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [CANDIDATE]: Term 1 election: Requested vote from peers 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359), 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025)
I20260925 01:29:10.865583 29857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f77dc287584455d96ee5447950d4212" candidate_uuid: "69af43f9aba046e393b5408cc90fa3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723d26a819f54a4d810288ae60733cd6"
I20260925 01:29:10.865600 29576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f77dc287584455d96ee5447950d4212" candidate_uuid: "69af43f9aba046e393b5408cc90fa3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484d2152b4524cddafa6faf8c16d71de"
I20260925 01:29:10.865805 29576 raft_consensus.cc:3037] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:10.865839 29857 raft_consensus.cc:3037] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:10.868003 29857 raft_consensus.cc:2454] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69af43f9aba046e393b5408cc90fa3d3 in term 1.
I20260925 01:29:10.868654 29645 leader_election.cc:302] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69af43f9aba046e393b5408cc90fa3d3, 723d26a819f54a4d810288ae60733cd6; no voters:
I20260925 01:29:10.868834 29576 raft_consensus.cc:2454] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69af43f9aba046e393b5408cc90fa3d3 in term 1.
I20260925 01:29:10.869154 29934 raft_consensus.cc:2781] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:10.869477 29934 raft_consensus.cc:686] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 LEADER]: Becoming Leader. State: Replica: 69af43f9aba046e393b5408cc90fa3d3, State: Running, Role: LEADER
I20260925 01:29:10.870056 29934 consensus_queue.cc:237] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.874125 29415 catalog_manager.cc:5495] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2). New cstate: current_term: 1 leader_uuid: "69af43f9aba046e393b5408cc90fa3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } health_report { overall_health: UNKNOWN } } }
Sep 25 01:29:10 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299748, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:10.961998 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.937090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33937 (local address 127.0.0.1:48024)
0925 01:29:10.938630 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:10.938658 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:10.938878 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:10.939691 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:10.939696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:10.940960 (+ 1264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:10.941540 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.941554 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.943792 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.943798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.945890 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:10.945905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.947449 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.947456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:10.947768 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.948803 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:10.948848 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:10.953050 (+ 4202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:10.958377 (+ 5327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:10.958402 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:10.958411 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:10.959237 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:10.959693 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:10.959698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:10.959701 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:10.960035 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:10.960721 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:10.960730 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:10.960967 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 01:29:10.961197 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 01:29:10.961521 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"thread_start_us":905,"threads_started":1}
I20260925 01:29:10.962096 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:10.937090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.27.160.62:33937)
0925 01:29:10.939202 (+ 2112us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:10.939219 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:10.939262 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:29:10.939367 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:10.939373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:10.939465 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:10.939598 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:10.941749 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.943613 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.946092 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:10.947204 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:10.947386 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:10.953424 (+ 6038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:10.953432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:10.953437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:10.953500 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:10.958191 (+ 4691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:10.959401 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:10.959407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:10.959410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:10.959575 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:10.960169 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:10.960174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:10.960177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:10.960563 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:10.960718 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:10.961331 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 01:29:10.961703 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":1917,"thread_start_us":1563,"threads_started":1}
I20260925 01:29:10.968514 29415 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:48024:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:29:10.969596 29415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:10.977569 29552 tablet_service.cc:1467] Processing CreateTablet for tablet 3075cb0fd5344a80bd2eac3569b32dbc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53d1e4a2fe1f46f698a88c65a5544b9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:29:10.977975 29692 tablet_service.cc:1467] Processing CreateTablet for tablet 3075cb0fd5344a80bd2eac3569b32dbc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53d1e4a2fe1f46f698a88c65a5544b9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:29:10.978233 29552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3075cb0fd5344a80bd2eac3569b32dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.978535 29692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3075cb0fd5344a80bd2eac3569b32dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.982455 29834 tablet_service.cc:1467] Processing CreateTablet for tablet 3075cb0fd5344a80bd2eac3569b32dbc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=53d1e4a2fe1f46f698a88c65a5544b9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:29:10.983034 29834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3075cb0fd5344a80bd2eac3569b32dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:10.985373 29923 tablet_bootstrap.cc:492] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: Bootstrap starting.
I20260925 01:29:10.986195 29921 tablet_bootstrap.cc:492] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: Bootstrap starting.
I20260925 01:29:10.987414 29923 tablet_bootstrap.cc:654] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.988220 29921 tablet_bootstrap.cc:654] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.995827 29922 tablet_bootstrap.cc:492] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: Bootstrap starting.
I20260925 01:29:10.997488 29921 tablet_bootstrap.cc:492] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: No bootstrap required, opened a new log
I20260925 01:29:10.997617 29923 tablet_bootstrap.cc:492] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: No bootstrap required, opened a new log
I20260925 01:29:10.997712 29921 ts_tablet_manager.cc:1397] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260925 01:29:10.997907 29923 ts_tablet_manager.cc:1397] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260925 01:29:10.998773 29922 tablet_bootstrap.cc:654] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:10.998620 29921 raft_consensus.cc:348] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.998939 29921 raft_consensus.cc:374] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:10.998888 29923 raft_consensus.cc:348] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.999071 29921 raft_consensus.cc:729] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69af43f9aba046e393b5408cc90fa3d3, State: Initialized, Role: FOLLOWER
I20260925 01:29:10.999257 29923 raft_consensus.cc:374] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:10.999418 29923 raft_consensus.cc:729] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484d2152b4524cddafa6faf8c16d71de, State: Initialized, Role: FOLLOWER
I20260925 01:29:10.999586 29921 consensus_queue.cc:260] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:10.999773 29923 consensus_queue.cc:260] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:11.000473 29923 ts_tablet_manager.cc:1428] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 01:29:11.000792 29934 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.001307 29934 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:11.000478 29929 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.003029 29929 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:11.008311 29921 ts_tablet_manager.cc:1428] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260925 01:29:11.011701 29922 tablet_bootstrap.cc:492] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: No bootstrap required, opened a new log
I20260925 01:29:11.011976 29922 ts_tablet_manager.cc:1397] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20260925 01:29:11.013157 29922 raft_consensus.cc:348] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:11.013597 29922 raft_consensus.cc:374] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:11.013761 29922 raft_consensus.cc:729] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723d26a819f54a4d810288ae60733cd6, State: Initialized, Role: FOLLOWER
I20260925 01:29:11.014147 29922 consensus_queue.cc:260] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:11.014742 29928 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.015082 29922 ts_tablet_manager.cc:1428] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 01:29:11.015190 29928 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:11.074335 29929 raft_consensus.cc:482] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:11.074641 29929 raft_consensus.cc:504] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:11.084285 29929 leader_election.cc:288] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2:44997), 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025)
I20260925 01:29:11.096791 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.077534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:40185 (local address 127.27.160.2:44997)
0925 01:29:11.078235 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.078241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.087351 (+ 9110us) server_negotiation.cc:407] Connection header received
0925 01:29:11.087543 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.087552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.087659 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.087773 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:11.088781 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.090309 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.093211 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.094739 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.094959 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.095050 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.096260 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.096421 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":406,"threads_started":1}
I20260925 01:29:11.097254 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.077093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:44997 (local address 127.27.160.1:40185)
0925 01:29:11.086882 (+ 9789us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:11.086930 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:11.087190 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:11.088069 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:11.088077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:11.088123 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:11.088510 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.088526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.090555 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.090563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.092989 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.093005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.095178 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.095184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.095392 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.095529 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 01:29:11.095685 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 01:29:11.095833 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9590,"mutex_wait_us":31,"thread_start_us":9388,"threads_started":1}
I20260925 01:29:11.097856 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3075cb0fd5344a80bd2eac3569b32dbc" candidate_uuid: "484d2152b4524cddafa6faf8c16d71de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69af43f9aba046e393b5408cc90fa3d3" is_pre_election: true
I20260925 01:29:11.098117 29717 raft_consensus.cc:2454] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484d2152b4524cddafa6faf8c16d71de in term 0.
I20260925 01:29:11.098881 29504 leader_election.cc:302] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d2152b4524cddafa6faf8c16d71de, 69af43f9aba046e393b5408cc90fa3d3; no voters:
I20260925 01:29:11.099334 29929 raft_consensus.cc:2781] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:11.099457 29929 raft_consensus.cc:482] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:11.099542 29929 raft_consensus.cc:3037] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:11.100944 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.078567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:44025 (local address 127.27.160.1:34887)
0925 01:29:11.089609 (+ 11042us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:11.089629 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:11.089733 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:11.092226 (+ 2493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:11.092231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:11.092255 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:11.092654 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.092666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.094441 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.094447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.098177 (+ 3730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.098192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.100056 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.100064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.100239 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.100282 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 01:29:11.100368 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:29:11.100510 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10912,"mutex_wait_us":55,"thread_start_us":10752,"threads_started":1}
I20260925 01:29:11.101670 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.078781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:34887 (local address 127.27.160.3:44025)
0925 01:29:11.091777 (+ 12996us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.091783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.091803 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:29:11.091868 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.091873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.091942 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.092040 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:11.092929 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.094253 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.098400 (+ 4147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.099646 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.099823 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.099930 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.101174 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.101337 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12839,"thread_start_us":13977,"threads_started":1}
I20260925 01:29:11.103005 29929 raft_consensus.cc:504] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:11.103286 29857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3075cb0fd5344a80bd2eac3569b32dbc" candidate_uuid: "484d2152b4524cddafa6faf8c16d71de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723d26a819f54a4d810288ae60733cd6" is_pre_election: true
I20260925 01:29:11.103559 29857 raft_consensus.cc:2454] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484d2152b4524cddafa6faf8c16d71de in term 0.
I20260925 01:29:11.104251 29929 leader_election.cc:288] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [CANDIDATE]: Term 1 election: Requested vote from peers 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2:44997), 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025)
I20260925 01:29:11.104249 29857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3075cb0fd5344a80bd2eac3569b32dbc" candidate_uuid: "484d2152b4524cddafa6faf8c16d71de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "723d26a819f54a4d810288ae60733cd6"
I20260925 01:29:11.104241 29717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3075cb0fd5344a80bd2eac3569b32dbc" candidate_uuid: "484d2152b4524cddafa6faf8c16d71de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69af43f9aba046e393b5408cc90fa3d3"
I20260925 01:29:11.104460 29857 raft_consensus.cc:3037] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:11.104480 29717 raft_consensus.cc:3037] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:11.107136 29717 raft_consensus.cc:2454] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484d2152b4524cddafa6faf8c16d71de in term 1.
I20260925 01:29:11.107722 29504 leader_election.cc:302] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484d2152b4524cddafa6faf8c16d71de, 69af43f9aba046e393b5408cc90fa3d3; no voters:
I20260925 01:29:11.107995 29857 raft_consensus.cc:2454] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484d2152b4524cddafa6faf8c16d71de in term 1.
I20260925 01:29:11.108075 29929 raft_consensus.cc:2781] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:11.108839 29929 raft_consensus.cc:686] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [term 1 LEADER]: Becoming Leader. State: Replica: 484d2152b4524cddafa6faf8c16d71de, State: Running, Role: LEADER
I20260925 01:29:11.109309 29929 consensus_queue.cc:237] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } }
I20260925 01:29:11.111621 29967 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: TxnStatusTablet state changed. Reason: New leader 484d2152b4524cddafa6faf8c16d71de. Latest consensus state: current_term: 1 leader_uuid: "484d2152b4524cddafa6faf8c16d71de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.112082 29967 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:29:11.113885 29414 catalog_manager.cc:5495] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de reported cstate change: term changed from 0 to 1, leader changed from <none> to 484d2152b4524cddafa6faf8c16d71de (127.27.160.1). New cstate: current_term: 1 leader_uuid: "484d2152b4524cddafa6faf8c16d71de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } health_report { overall_health: UNKNOWN } } }
I20260925 01:29:11.114393 29969 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:29:11.114825 29969 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:29:11.148895 29959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.138767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46359 (local address 127.0.0.1:52134)
0925 01:29:11.139096 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:11.139122 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:11.139289 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:11.141784 (+ 2495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:11.141790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:11.141804 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:29:11.141993 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.142004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.143839 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.143845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.145510 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.145528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.146645 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.146651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.146876 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.146960 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:29:11.148244 (+ 1284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:29:11.148270 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:29:11.148374 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:29:11.148549 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 01:29:11.149289 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.139814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.27.160.1:46359)
0925 01:29:11.141255 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.141262 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.141291 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:29:11.141399 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.141405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.141528 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.141745 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:11.142179 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.143652 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.145682 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.146457 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.146603 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.147230 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:11.148108 (+ 878us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:11.148280 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.148415 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.148511 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":611,"threads_started":1}
I20260925 01:29:11.156034 29857 raft_consensus.cc:1261] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Refusing update from remote peer 484d2152b4524cddafa6faf8c16d71de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:11.156659 29717 raft_consensus.cc:1261] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Refusing update from remote peer 484d2152b4524cddafa6faf8c16d71de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:11.157212 29967 consensus_queue.cc:1035] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [LEADER]: Connected to new peer: Peer: permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:11.157955 29929 consensus_queue.cc:1035] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de [LEADER]: Connected to new peer: Peer: permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:11.181077 29928 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: TxnStatusTablet state changed. Reason: New leader 484d2152b4524cddafa6faf8c16d71de. Latest consensus state: current_term: 1 leader_uuid: "484d2152b4524cddafa6faf8c16d71de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.181658 29928 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:11.191694 29934 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: TxnStatusTablet state changed. Reason: New leader 484d2152b4524cddafa6faf8c16d71de. Latest consensus state: current_term: 1 leader_uuid: "484d2152b4524cddafa6faf8c16d71de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.192050 29934 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:11.195629 29928 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "484d2152b4524cddafa6faf8c16d71de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.198468 29928 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 723d26a819f54a4d810288ae60733cd6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:11.202574 29929 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "484d2152b4524cddafa6faf8c16d71de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.203001 29929 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:29:11.206664 29929 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "484d2152b4524cddafa6faf8c16d71de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.206983 29929 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 484d2152b4524cddafa6faf8c16d71de: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:29:11.210911 29934 tablet_replica.cc:439] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "484d2152b4524cddafa6faf8c16d71de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 } } peers { permanent_uuid: "69af43f9aba046e393b5408cc90fa3d3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 44997 } } peers { permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 } } }
I20260925 01:29:11.211233 29934 tablet_replica.cc:442] T 3075cb0fd5344a80bd2eac3569b32dbc P 69af43f9aba046e393b5408cc90fa3d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:11.238968 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.228338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:41501 (local address 127.27.160.1:46359)
0925 01:29:11.228567 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.228574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.229938 (+ 1364us) server_negotiation.cc:407] Connection header received
0925 01:29:11.230173 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.230179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.230284 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.230380 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:11.231488 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.233291 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.236345 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.237306 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.237430 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.238053 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:11.238189 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:11.238274 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.238437 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.238519 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:29:11.239055 29983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.228359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46359 (local address 127.27.160.2:41501)
0925 01:29:11.229784 (+ 1425us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:11.229806 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:11.229981 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:11.230436 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:11.230442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:11.230467 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:29:11.231237 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.231252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.233512 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.233519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.236148 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.236167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.237542 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.237549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.237736 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.237813 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:29:11.238301 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:29:11.238314 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 01:29:11.238399 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 01:29:11.238544 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"thread_start_us":773,"threads_started":1}
I20260925 01:29:11.256618 29856 raft_consensus.cc:1261] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Refusing update from remote peer 69af43f9aba046e393b5408cc90fa3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:11.256703 29576 raft_consensus.cc:1261] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Refusing update from remote peer 69af43f9aba046e393b5408cc90fa3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:11.257313 29927 consensus_queue.cc:1035] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "723d26a819f54a4d810288ae60733cd6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:11.258119 29934 consensus_queue.cc:1035] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "484d2152b4524cddafa6faf8c16d71de" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:11.312415 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.304210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:58225 (local address 127.27.160.2:44997)
0925 01:29:11.304443 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.304449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.305147 (+ 698us) server_negotiation.cc:407] Connection header received
0925 01:29:11.305891 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.305898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.305992 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.306099 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:11.306808 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.307918 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.310231 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.310883 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.311044 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.311564 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:11.311691 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:11.311803 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.312006 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.312095 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:29:11.312610 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.304031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:44997 (local address 127.27.160.1:58225)
0925 01:29:11.304996 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:11.305018 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:11.305236 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:11.306129 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:11.306134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:11.306146 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:29:11.306619 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.306634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.308092 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.308097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.310088 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:11.310105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.311098 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.311109 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:11.311278 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.311344 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:29:11.311864 (+ 520us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:29:11.311874 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:29:11.311980 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 01:29:11.312142 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":428,"threads_started":1}
I20260925 01:29:11.357254 29415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:11.357659 29415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:11.361266 29415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c322196f33843389915b23a72ba9d9a: Sending DeleteTablet for 3 replicas of tablet 0f77dc287584455d96ee5447950d4212
I20260925 01:29:11.362275 29552 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f77dc287584455d96ee5447950d4212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:11 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:57672
I20260925 01:29:11.362340 29692 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f77dc287584455d96ee5447950d4212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:11 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:52500
I20260925 01:29:11.362397 29834 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f77dc287584455d96ee5447950d4212 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:11 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:48722
I20260925 01:29:11.374343 29994 tablet_replica.cc:330] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: stopping tablet replica
I20260925 01:29:11.374699 29994 raft_consensus.cc:2229] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:11.375983 29993 tablet_replica.cc:330] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: stopping tablet replica
I20260925 01:29:11.376276 29993 raft_consensus.cc:2229] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:11.376963 29993 raft_consensus.cc:2258] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:11.377585 29994 raft_consensus.cc:2258] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:11.378966 29994 ts_tablet_manager.cc:1905] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:11.380815 29992 tablet_replica.cc:330] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: stopping tablet replica
I20260925 01:29:11.381640 29993 ts_tablet_manager.cc:1905] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:11.382205 29992 raft_consensus.cc:2229] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:11.383092 29992 raft_consensus.cc:2258] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:11.385258 29992 ts_tablet_manager.cc:1905] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:11.388257 29994 ts_tablet_manager.cc:1918] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:11.388418 29994 log.cc:1196] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/0f77dc287584455d96ee5447950d4212
I20260925 01:29:11.388998 29994 ts_tablet_manager.cc:1939] T 0f77dc287584455d96ee5447950d4212 P 723d26a819f54a4d810288ae60733cd6: Deleting consensus metadata
I20260925 01:29:11.389856 29993 ts_tablet_manager.cc:1918] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:11.390043 29993 log.cc:1196] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/0f77dc287584455d96ee5447950d4212
I20260925 01:29:11.390309 29402 catalog_manager.cc:4841] TS 723d26a819f54a4d810288ae60733cd6 (127.27.160.3:44025): tablet 0f77dc287584455d96ee5447950d4212 (table test-table [id=b60b82baacd146dc8d0f993a3aa93fe2]) successfully deleted
I20260925 01:29:11.390532 29993 ts_tablet_manager.cc:1939] T 0f77dc287584455d96ee5447950d4212 P 69af43f9aba046e393b5408cc90fa3d3: Deleting consensus metadata
I20260925 01:29:11.391765 29401 catalog_manager.cc:4841] TS 69af43f9aba046e393b5408cc90fa3d3 (127.27.160.2:44997): tablet 0f77dc287584455d96ee5447950d4212 (table test-table [id=b60b82baacd146dc8d0f993a3aa93fe2]) successfully deleted
I20260925 01:29:11.397143 29992 ts_tablet_manager.cc:1918] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:11.397333 29992 log.cc:1196] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/0f77dc287584455d96ee5447950d4212
I20260925 01:29:11.397871 29992 ts_tablet_manager.cc:1939] T 0f77dc287584455d96ee5447950d4212 P 484d2152b4524cddafa6faf8c16d71de: Deleting consensus metadata
I20260925 01:29:11.399108 29401 catalog_manager.cc:4841] TS 484d2152b4524cddafa6faf8c16d71de (127.27.160.1:46359): tablet 0f77dc287584455d96ee5447950d4212 (table test-table [id=b60b82baacd146dc8d0f993a3aa93fe2]) successfully deleted
Sep 25 01:29:11 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:11.410673 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.378935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.27.160.1:46359)
0925 01:29:11.379316 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.379334 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.386818 (+ 7484us) server_negotiation.cc:407] Connection header received
0925 01:29:11.388875 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.388881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.389019 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.389152 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:11.392675 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.394062 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.396069 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.396890 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.397034 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.402434 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:11.402453 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:11.402477 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:11.402534 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:11.407243 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.408122 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.408133 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.408137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.408302 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.408718 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.408724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.408727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.409255 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:11.409595 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.409953 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.410209 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 01:29:11.411672 29617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52148
I20260925 01:29:11.431360 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.415375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.27.160.62:33937)
0925 01:29:11.415613 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.415619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.415851 (+ 232us) server_negotiation.cc:407] Connection header received
0925 01:29:11.415993 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.415999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.416100 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.416210 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:11.417529 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.418842 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.420226 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.421034 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.421247 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.423993 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:11.424003 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:11.424009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:11.424058 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:11.428066 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.428848 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.428855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.428859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.429068 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.429582 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.429591 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.429597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.429988 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:11.430252 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.430708 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.430973 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 01:29:11.432122 29415 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48040
I20260925 01:29:11.434661 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29486
I20260925 01:29:11.452653 29972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.435680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.27.160.1:46359)
0925 01:29:11.435891 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.435899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.436154 (+ 255us) server_negotiation.cc:407] Connection header received
0925 01:29:11.436291 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.436297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.436404 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.436528 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:11.437756 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.438964 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.440269 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.441169 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.441313 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.445037 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:11.445043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:11.445046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:11.445082 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:11.448458 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.449668 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.449672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.449674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.449900 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.450507 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.450510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.450513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.450960 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:11.451280 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.451764 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.452094 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 01:29:11.453593 29617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52162
W20260925 01:29:11.454530 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29486: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:11.454914 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 29486
I20260925 01:29:11.475600 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29626
Sep 25 01:29:11 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:29:11.492211 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.476468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.27.160.2:44997)
0925 01:29:11.476638 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.476643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.476717 (+ 74us) server_negotiation.cc:407] Connection header received
0925 01:29:11.476909 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.476913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.476998 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.477084 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:11.478323 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.479227 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.480445 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.481105 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.481278 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.486219 (+ 4941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:11.486237 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:11.486254 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:11.486364 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:11.489439 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.490333 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.490350 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.490353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.490495 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.490818 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.490821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.490823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.491268 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:11.491530 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.491762 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.491921 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 01:29:11.492970 29757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50592
W20260925 01:29:11.493769 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29626: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:11.493958 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 29626
I20260925 01:29:11.515910 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29766
Sep 25 01:29:11 dist-test-slave-h6hf krb5kdc[29370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:29:11.533182 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:11.516739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.27.160.3:44025)
0925 01:29:11.516972 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:11.516979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:11.517149 (+ 170us) server_negotiation.cc:407] Connection header received
0925 01:29:11.517410 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:11.517415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:11.517478 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:11.517599 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:11.518887 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.519577 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.520811 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:11.521343 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:11.521483 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:11.526173 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:11.526191 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:11.526208 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:11.526246 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:11.529036 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.530117 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.530129 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.530135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.530325 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:11.531011 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:11.531018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:11.531024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:11.531561 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:11.531935 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:11.532399 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 01:29:11.532714 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 01:29:11.534003 29897 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54486
W20260925 01:29:11.534713 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29766: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:11.534878 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 29766
I20260925 01:29:11.554399 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 29383
I20260925 01:29:11.555747 29455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:11.673847 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 29383
2026-09-25T01:29:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5231 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:11 dist-test-slave-h6hf krb5kdc[30003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:11 dist-test-slave-h6hf krb5kdc[30003](info): set up 2 sockets
Sep 25 01:29:11 dist-test-slave-h6hf krb5kdc[30003](info): commencing operation
krb5kdc: starting...
W20260925 01:29:12.790875 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:12 dist-test-slave-h6hf krb5kdc[30003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:12.990365 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44805
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:13.328189 30016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:13.328554 30016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:13.328738 30016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:13.336054 30016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:13.336212 30016 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:29:13.336313 30016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:13.336426 30016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:13.336519 30016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:13.336607 30016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:13.346603 30016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:44805
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:13.347983 30016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:13.349377 30016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:13.361784 30021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:13.361807 30022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:13.361739 30024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:13.363145 30016 server_base.cc:1034] running on GCE node
I20260925 01:29:13.363983 30016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:13.365696 30016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:13.366935 30016 hybrid_clock.cc:648] HybridClock initialized: now 1790299753366868 us; error 36 us; skew 500 ppm
I20260925 01:29:13.367419 30016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:13 dist-test-slave-h6hf krb5kdc[30003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299753, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:13.370734 30016 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:13.372319 30016 webserver.cc:460] Webserver started at http://127.27.160.62:45119/ using document root <none> and password file <none>
I20260925 01:29:13.373044 30016 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:13.373179 30016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:13.373490 30016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:13.376523 30016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "6288b8aa0da74e91a99507448d218b5c"
format_stamp: "Formatted at 2026-09-25 01:29:13 on dist-test-slave-h6hf"
server_key: "2c47b5c0042ea3c614d0f58230d43a2e"
server_key_iv: "43d300a5eaa13cdef13719c675ab3086"
server_key_version: "encryptionkey@0"
I20260925 01:29:13.377327 30016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "6288b8aa0da74e91a99507448d218b5c"
format_stamp: "Formatted at 2026-09-25 01:29:13 on dist-test-slave-h6hf"
server_key: "2c47b5c0042ea3c614d0f58230d43a2e"
server_key_iv: "43d300a5eaa13cdef13719c675ab3086"
server_key_version: "encryptionkey@0"
I20260925 01:29:13.382726 30016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:29:13.386730 30031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:13.388381 30016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:13.388554 30016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "6288b8aa0da74e91a99507448d218b5c"
format_stamp: "Formatted at 2026-09-25 01:29:13 on dist-test-slave-h6hf"
server_key: "2c47b5c0042ea3c614d0f58230d43a2e"
server_key_iv: "43d300a5eaa13cdef13719c675ab3086"
server_key_version: "encryptionkey@0"
I20260925 01:29:13.388777 30016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:13.412102 30016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:13.416966 30016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:13.417372 30016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:13.442119 30016 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:44805
I20260925 01:29:13.442139 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:44805 every 8 connection(s)
I20260925 01:29:13.443704 30016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:13.448074 30093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:13.449790 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 30016
I20260925 01:29:13.450026 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:13.450958 28288 external_mini_cluster.cc:1428] Setting key 066d9fea2e0489ec3efadfa81afe1004
I20260925 01:29:13.463095 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: Bootstrap starting.
Sep 25 01:29:13 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:13.467259 30093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:13.468412 30093 log.cc:824] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:13.471649 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: No bootstrap required, opened a new log
I20260925 01:29:13.475765 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:13.453501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.27.160.62:44805)
0925 01:29:13.454485 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:13.454498 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:13.454547 (+ 49us) server_negotiation.cc:407] Connection header received
0925 01:29:13.456360 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:13.456390 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:13.456853 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:13.457374 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:13.459085 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:13.460601 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:13.462121 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:13.462907 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:13.463139 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:13.467971 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:13.467980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:13.467996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:13.468035 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:13.471547 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:13.472319 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:13.472326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:13.472333 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:13.472499 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:13.472898 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:13.472902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:13.472904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:13.473502 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:13.473831 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:13.474296 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 01:29:13.474735 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":467,"threads_started":1}
I20260925 01:29:13.481073 30093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288b8aa0da74e91a99507448d218b5c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44805 } }
I20260925 01:29:13.481586 30093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:13.481715 30093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6288b8aa0da74e91a99507448d218b5c, State: Initialized, Role: FOLLOWER
I20260925 01:29:13.482370 30093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288b8aa0da74e91a99507448d218b5c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44805 } }
I20260925 01:29:13.482647 30093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:13.482803 30093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:13.483002 30093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:13.485458 30093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288b8aa0da74e91a99507448d218b5c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44805 } }
I20260925 01:29:13.486027 30093 leader_election.cc:302] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6288b8aa0da74e91a99507448d218b5c; no voters:
I20260925 01:29:13.486709 30093 leader_election.cc:288] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:13.486852 30098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:13.488207 30098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [term 1 LEADER]: Becoming Leader. State: Replica: 6288b8aa0da74e91a99507448d218b5c, State: Running, Role: LEADER
I20260925 01:29:13.488850 30098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288b8aa0da74e91a99507448d218b5c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44805 } }
I20260925 01:29:13.489291 30093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:13.491606 30099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6288b8aa0da74e91a99507448d218b5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288b8aa0da74e91a99507448d218b5c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44805 } } }
I20260925 01:29:13.491976 30099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:13.492491 30100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6288b8aa0da74e91a99507448d218b5c. Latest consensus state: current_term: 1 leader_uuid: "6288b8aa0da74e91a99507448d218b5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288b8aa0da74e91a99507448d218b5c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44805 } } }
I20260925 01:29:13.492823 30100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:13.495152 30104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:13.500156 30104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:13.510150 30104 catalog_manager.cc:1347] Generated new cluster ID: f6cff5ebbc434999ba1cd53e5a8dce49
I20260925 01:29:13.510285 30104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:13.541292 30104 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:13.542713 30104 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:13.552232 30104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: Generated new TSK 0
I20260925 01:29:13.553210 30104 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:29:13.610360 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44805
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:13.967700 30120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:13.968068 30120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:13.968253 30120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:13.976007 30120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:13.976193 30120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:13.976368 30120 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:29:13.976472 30120 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:29:13.976624 30120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:13.987185 30120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:44805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:13.988699 30120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:13.990274 30120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:14.004650 30125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:14.004680 30128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:14.004704 30126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:14.006243 30120 server_base.cc:1034] running on GCE node
I20260925 01:29:14.007031 30120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:14.008103 30120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:14.009377 30120 hybrid_clock.cc:648] HybridClock initialized: now 1790299754009316 us; error 52 us; skew 500 ppm
I20260925 01:29:14.009750 30120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:14 dist-test-slave-h6hf krb5kdc[30003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299754, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:14.017915 30120 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:29:14.019641 30120 webserver.cc:460] Webserver started at http://127.27.160.1:41853/ using document root <none> and password file <none>
I20260925 01:29:14.020381 30120 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:14.020502 30120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:14.020789 30120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:14.024153 30120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "013664afced241cd81da058a46acb28a"
format_stamp: "Formatted at 2026-09-25 01:29:14 on dist-test-slave-h6hf"
server_key: "ae8fe3d79ea8c01d16fb0217805ed1c6"
server_key_iv: "817f66077a143056a2da9e48bc6a88a4"
server_key_version: "encryptionkey@0"
I20260925 01:29:14.024945 30120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "013664afced241cd81da058a46acb28a"
format_stamp: "Formatted at 2026-09-25 01:29:14 on dist-test-slave-h6hf"
server_key: "ae8fe3d79ea8c01d16fb0217805ed1c6"
server_key_iv: "817f66077a143056a2da9e48bc6a88a4"
server_key_version: "encryptionkey@0"
I20260925 01:29:14.030717 30120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:29:14.034731 30135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:14.036262 30120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:29:14.036444 30120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "013664afced241cd81da058a46acb28a"
format_stamp: "Formatted at 2026-09-25 01:29:14 on dist-test-slave-h6hf"
server_key: "ae8fe3d79ea8c01d16fb0217805ed1c6"
server_key_iv: "817f66077a143056a2da9e48bc6a88a4"
server_key_version: "encryptionkey@0"
I20260925 01:29:14.036646 30120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:14.076395 30120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:14.080519 30120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:14.080845 30120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:14.083673 30120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:14.083771 30120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:14.083914 30120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:14.084024 30120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:29:14 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299754, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:14.111991 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:14.084661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:58011 (local address 127.27.160.62:44805)
0925 01:29:14.085174 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:14.085180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:14.086533 (+ 1353us) server_negotiation.cc:407] Connection header received
0925 01:29:14.090231 (+ 3698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:14.090240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:14.090327 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:14.090467 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:14.093275 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.094104 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.097464 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.097954 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.098135 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:14.103699 (+ 5564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:14.103710 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:14.103718 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:14.103763 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:14.107972 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:14.109210 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:14.109217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:14.109221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:14.109410 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:14.110019 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:14.110025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:14.110029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:14.110326 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:14.110518 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:14.111334 (+ 816us) server_negotiation.cc:299] Negotiation successful
0925 01:29:14.111590 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260925 01:29:14.113186 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:14.085418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44805 (local address 127.27.160.1:58011)
0925 01:29:14.086320 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:14.086384 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:14.089870 (+ 3486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:14.090769 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:14.090786 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:14.091853 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:14.093006 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:14.093044 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.094287 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.094299 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:14.097288 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:14.097306 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.098334 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.098346 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:14.098781 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:14.099779 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:14.099822 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:14.103406 (+ 3584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:14.108192 (+ 4786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:14.108209 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:14.108230 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:14.109023 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:14.109582 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:14.109594 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:14.109598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:14.109870 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:14.110491 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:14.110500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:14.110929 (+ 429us) client_negotiation.cc:769] Sending connection context
0925 01:29:14.111220 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 01:29:14.111809 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":347,"threads_started":1}
I20260925 01:29:14.135223 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:14.119508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:57629 (local address 127.27.160.62:44805)
0925 01:29:14.119683 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:14.119688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:14.120682 (+ 994us) server_negotiation.cc:407] Connection header received
0925 01:29:14.120997 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:14.121002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:14.121068 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:14.121239 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:14.122632 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.123761 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.125846 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.126320 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.126496 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:14.128484 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:14.128489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:14.128493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:14.128529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:14.131635 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:14.132694 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:14.132700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:14.132706 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:14.132896 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:14.133356 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:14.133361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:14.133365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:14.133802 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:14.134026 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:14.134536 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 01:29:14.134773 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 01:29:14.135316 30211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:14.119529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44805 (local address 127.27.160.1:57629)
0925 01:29:14.120546 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:14.120572 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:14.120819 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:14.121270 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:14.121275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:14.121982 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:14.122468 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:14.122481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.123970 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.123974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:14.125681 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:14.125696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.126548 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.126554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:14.126720 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:14.127428 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:14.127447 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:14.128252 (+ 805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:14.131784 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:14.131792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:14.131797 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:14.132513 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:14.133039 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:14.133043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:14.133045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:14.133236 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:14.133949 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:14.133952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:14.134233 (+ 281us) client_negotiation.cc:769] Sending connection context
0925 01:29:14.134422 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 01:29:14.134809 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":424,"threads_started":1}
I20260925 01:29:14.149315 30120 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:46579
I20260925 01:29:14.149354 30255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:46579 every 8 connection(s)
I20260925 01:29:14.150710 30120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:14.153384 30256 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44805
I20260925 01:29:14.153764 30256 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:14.154459 30256 heartbeater.cc:511] Master 127.27.160.62:44805 requested a full tablet report, sending...
I20260925 01:29:14.156507 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 30120
I20260925 01:29:14.156728 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:14.156788 30048 ts_manager.cc:194] Registered new tserver with Master: 013664afced241cd81da058a46acb28a (127.27.160.1:46579)
I20260925 01:29:14.157557 28288 external_mini_cluster.cc:1428] Setting key 84a5c9fdb482ea373cd1283daa74fbec
I20260925 01:29:14.158844 30048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:58011
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:14.200762 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44805
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:14.540762 30260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:14.541306 30260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:14.541522 30260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:14.549089 30260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:14.549292 30260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:14.549479 30260 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:29:14.549579 30260 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:29:14.549692 30260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:14.560041 30260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:44805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:14.561477 30260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:14.562899 30260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:14.576812 30265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:14.576905 30266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:14.576936 30268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:14.578040 30260 server_base.cc:1034] running on GCE node
I20260925 01:29:14.578786 30260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:14.579843 30260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:14.581092 30260 hybrid_clock.cc:648] HybridClock initialized: now 1790299754581046 us; error 47 us; skew 500 ppm
I20260925 01:29:14.581521 30260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:14 dist-test-slave-h6hf krb5kdc[30003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299754, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:14.584870 30260 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:14.586678 30260 webserver.cc:460] Webserver started at http://127.27.160.2:43447/ using document root <none> and password file <none>
I20260925 01:29:14.587399 30260 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:14.587560 30260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:14.587899 30260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:14.591055 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "bb5781d682be4d96924dae671f6e68ce"
format_stamp: "Formatted at 2026-09-25 01:29:14 on dist-test-slave-h6hf"
server_key: "8e9941e41a8c4a7b2bec780e3882f19e"
server_key_iv: "a3cb792f508c5ccd4c52419b3f5626e9"
server_key_version: "encryptionkey@0"
I20260925 01:29:14.591838 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "bb5781d682be4d96924dae671f6e68ce"
format_stamp: "Formatted at 2026-09-25 01:29:14 on dist-test-slave-h6hf"
server_key: "8e9941e41a8c4a7b2bec780e3882f19e"
server_key_iv: "a3cb792f508c5ccd4c52419b3f5626e9"
server_key_version: "encryptionkey@0"
I20260925 01:29:14.597455 30260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:29:14.601274 30275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:14.602699 30260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:14.602885 30260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "bb5781d682be4d96924dae671f6e68ce"
format_stamp: "Formatted at 2026-09-25 01:29:14 on dist-test-slave-h6hf"
server_key: "8e9941e41a8c4a7b2bec780e3882f19e"
server_key_iv: "a3cb792f508c5ccd4c52419b3f5626e9"
server_key_version: "encryptionkey@0"
I20260925 01:29:14.603094 30260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:14.637633 30260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:14.642789 30260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:14.643124 30260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:14.646356 30260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:14.646507 30260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:14.646657 30260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:14.646754 30260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:29:14 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299754, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:14.672999 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:14.646832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:34257 (local address 127.27.160.62:44805)
0925 01:29:14.647746 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:14.647751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:14.648817 (+ 1066us) server_negotiation.cc:407] Connection header received
0925 01:29:14.651584 (+ 2767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:14.651590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:14.651703 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:14.651836 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:14.654536 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.656091 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.658597 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.659161 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.659327 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:14.665074 (+ 5747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:14.665080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:14.665083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:14.665172 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:14.668680 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:14.669698 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:14.669706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:14.669711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:14.669979 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:14.670691 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:14.670697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:14.670700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:14.671135 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:14.671291 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:14.672351 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0925 01:29:14.672593 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":710,"threads_started":1}
I20260925 01:29:14.674163 30285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:14.647694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44805 (local address 127.27.160.2:34257)
0925 01:29:14.648612 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:14.648665 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:14.651253 (+ 2588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:14.652171 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:14.652190 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:14.653221 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:14.654235 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:14.654282 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.656314 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.656318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:14.658287 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:14.658304 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.659478 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.659486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:14.659813 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:14.660851 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:14.660900 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:14.664727 (+ 3827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:14.668860 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:14.668871 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:14.668891 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:14.669484 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:14.670123 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:14.670128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:14.670132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:14.670531 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:14.671259 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:14.671269 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:14.671937 (+ 668us) client_negotiation.cc:769] Sending connection context
0925 01:29:14.672252 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 01:29:14.672867 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":393,"threads_started":1}
I20260925 01:29:14.693850 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:14.680001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:45831 (local address 127.27.160.62:44805)
0925 01:29:14.680256 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:14.680266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:14.680829 (+ 563us) server_negotiation.cc:407] Connection header received
0925 01:29:14.680987 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:14.680996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:14.681114 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:14.681290 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:14.682630 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.683386 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.684732 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.685216 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.685352 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:14.687446 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:14.687454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:14.687463 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:14.687508 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:14.690974 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:14.691666 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:14.691673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:14.691676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:14.691802 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:14.692166 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:14.692172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:14.692175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:14.692556 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:14.692711 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:14.693250 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 01:29:14.693482 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:29:14.694190 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:14.680059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44805 (local address 127.27.160.2:45831)
0925 01:29:14.680716 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:14.680739 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:14.680854 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:14.681332 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:14.681339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:14.681996 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:14.682459 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:14.682470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.683531 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.683536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:14.684589 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:14.684600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:14.685464 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:14.685474 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:14.685744 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:14.686498 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:14.686515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:14.687228 (+ 713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:14.691123 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:14.691129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:14.691134 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:14.691537 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:14.691924 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:14.691928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:14.691932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:14.692060 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:14.692692 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:14.692697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:14.692961 (+ 264us) client_negotiation.cc:769] Sending connection context
0925 01:29:14.693178 (+ 217us) client_negotiation.cc:240] Negotiation successful
0925 01:29:14.693723 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":353,"threads_started":1}
I20260925 01:29:14.711002 30260 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:40597
I20260925 01:29:14.711031 30395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:40597 every 8 connection(s)
I20260925 01:29:14.712458 30260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:14.714783 30396 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44805
I20260925 01:29:14.715087 30396 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:14.715824 30396 heartbeater.cc:511] Master 127.27.160.62:44805 requested a full tablet report, sending...
I20260925 01:29:14.717387 30048 ts_manager.cc:194] Registered new tserver with Master: bb5781d682be4d96924dae671f6e68ce (127.27.160.2:40597)
I20260925 01:29:14.718230 30048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:34257
I20260925 01:29:14.721035 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 30260
I20260925 01:29:14.721241 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:14.721684 28288 external_mini_cluster.cc:1428] Setting key a4b36bce30a6605101c6522412a8dbb4
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:14.764901 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44805
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:15.092356 30400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:15.092648 30400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:15.092816 30400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:15.100260 30400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:15.100427 30400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:15.100589 30400 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:29:15.100679 30400 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:29:15.100819 30400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:15.111509 30400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:44805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:15.113014 30400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:15.114513 30400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:15.128094 30405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:15.128041 30406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:15.128005 30408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:15.128712 30400 server_base.cc:1034] running on GCE node
I20260925 01:29:15.129495 30400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:15.130467 30400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:15.131670 30400 hybrid_clock.cc:648] HybridClock initialized: now 1790299755131614 us; error 46 us; skew 500 ppm
I20260925 01:29:15.132027 30400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:15 dist-test-slave-h6hf krb5kdc[30003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299755, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:15.135344 30400 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:15.136868 30400 webserver.cc:460] Webserver started at http://127.27.160.3:35103/ using document root <none> and password file <none>
I20260925 01:29:15.137677 30400 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:15.137835 30400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:15.138150 30400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:15.141247 30400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "7e04f723ffc84d19a9b773129d6d93a6"
format_stamp: "Formatted at 2026-09-25 01:29:15 on dist-test-slave-h6hf"
server_key: "851c06daaa9448cb2faf03b9f3b274fb"
server_key_iv: "c1930e33c98c11f02344015727eabcfd"
server_key_version: "encryptionkey@0"
I20260925 01:29:15.142073 30400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "7e04f723ffc84d19a9b773129d6d93a6"
format_stamp: "Formatted at 2026-09-25 01:29:15 on dist-test-slave-h6hf"
server_key: "851c06daaa9448cb2faf03b9f3b274fb"
server_key_iv: "c1930e33c98c11f02344015727eabcfd"
server_key_version: "encryptionkey@0"
I20260925 01:29:15.147289 30400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 01:29:15.150980 30415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:15.152364 30400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:29:15.152546 30400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "7e04f723ffc84d19a9b773129d6d93a6"
format_stamp: "Formatted at 2026-09-25 01:29:15 on dist-test-slave-h6hf"
server_key: "851c06daaa9448cb2faf03b9f3b274fb"
server_key_iv: "c1930e33c98c11f02344015727eabcfd"
server_key_version: "encryptionkey@0"
I20260925 01:29:15.152752 30400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:15.161988 30256 heartbeater.cc:503] Master 127.27.160.62:44805 was elected leader, sending a full tablet report...
I20260925 01:29:15.184319 30400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:15.188941 30400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:15.189311 30400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:15.192240 30400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:15.192375 30400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:15.192513 30400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:15.192646 30400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:29:15 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299755, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:15.218498 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.193372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:56593 (local address 127.27.160.62:44805)
0925 01:29:15.193575 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.193581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.195251 (+ 1670us) server_negotiation.cc:407] Connection header received
0925 01:29:15.198517 (+ 3266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.198523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.198639 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.198756 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:15.201343 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.202446 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.205643 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.206409 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.206556 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.210679 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:15.210684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:15.210687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:15.210745 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:15.214542 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:15.215639 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:15.215644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:15.215648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:15.215837 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:15.216515 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:15.216521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:15.216526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:15.216829 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:15.216944 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.218063 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.218225 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:29:15.220402 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.194038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44805 (local address 127.27.160.3:56593)
0925 01:29:15.195014 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.195089 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.198107 (+ 3018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.199099 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.199116 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.200061 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:15.201089 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.201148 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.202641 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.202653 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.205447 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.205465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.206734 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.206744 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.207039 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.207867 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:15.207893 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:15.210453 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:15.214712 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:15.214726 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:15.214747 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:15.215522 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:15.215979 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:15.215984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:15.215988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:15.216362 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:15.216957 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:15.216969 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:15.217562 (+ 593us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.217965 (+ 403us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.218691 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":373,"threads_started":1}
I20260925 01:29:15.244207 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.228479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44805 (local address 127.27.160.3:45407)
0925 01:29:15.229189 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.229217 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.229386 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.229736 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.229741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.230225 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:15.230511 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.230520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.231547 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.231552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.232571 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.232583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.233465 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.233471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.233654 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.234399 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:15.234424 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:15.235709 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:15.240352 (+ 4643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:15.240358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:15.240362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:15.241483 (+ 1121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:15.241897 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:15.241902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:15.241906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:15.242078 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:15.242662 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:15.242668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:15.242947 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.243180 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.243570 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":383,"threads_started":1}
I20260925 01:29:15.245069 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.228437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:45407 (local address 127.27.160.62:44805)
0925 01:29:15.228675 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.228683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.229306 (+ 623us) server_negotiation.cc:407] Connection header received
0925 01:29:15.229531 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.229536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.229610 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.229710 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:15.230657 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.231407 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.232714 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.233241 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.233381 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.236184 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:15.236191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:15.236196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:15.236242 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:15.240104 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:15.241611 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:15.241616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:15.241619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:15.241764 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:15.242201 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:15.242205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:15.242208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:15.242525 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:15.242700 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.244584 (+ 1884us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.244804 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:29:15.254627 30400 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:40803
I20260925 01:29:15.254648 30534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:40803 every 8 connection(s)
I20260925 01:29:15.255851 30400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:15.258285 30535 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44805
I20260925 01:29:15.258579 30535 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:15.259346 30535 heartbeater.cc:511] Master 127.27.160.62:44805 requested a full tablet report, sending...
I20260925 01:29:15.260877 30048 ts_manager.cc:194] Registered new tserver with Master: 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:15.261825 30048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:56593
I20260925 01:29:15.264684 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 30400
I20260925 01:29:15.264855 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:15.265403 28288 external_mini_cluster.cc:1428] Setting key af362cf080be62e105852993d9985ed1
I20260925 01:29:15.268386 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:29:15 dist-test-slave-h6hf krb5kdc[30003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:29:15 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:15.320060 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.304957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44336 (local address 127.27.160.62:44805)
0925 01:29:15.305246 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.305252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.305528 (+ 276us) server_negotiation.cc:407] Connection header received
0925 01:29:15.305738 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.305743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.305816 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.305926 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:15.307445 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.308430 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.309724 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.310175 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.310292 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.314285 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:15.314291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:15.314295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:15.314400 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:15.317689 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:15.318313 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:15.318317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:15.318320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:15.318489 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:15.318840 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:15.318843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:15.318846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:15.319156 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:15.319367 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.319586 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.319767 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 01:29:15.324425 30048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:15.331991 30048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:15.356956 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.345501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.27.160.1:46579)
0925 01:29:15.348442 (+ 2941us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.348451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.348484 (+ 33us) server_negotiation.cc:407] Connection header received
0925 01:29:15.348601 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.348610 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.348861 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.349097 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:15.349953 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.351519 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.354016 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.355490 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.355682 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.355941 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.356426 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.356620 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2774,"thread_start_us":2621,"threads_started":1}
I20260925 01:29:15.357079 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.345277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.0.0.1:39892)
0925 01:29:15.346130 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.346198 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.346410 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.349097 (+ 2687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.349106 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.349164 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:15.349741 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.349757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.351675 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.351681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.353759 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.353771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.355709 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.355716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.355999 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.356078 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.356222 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.356471 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":5,"thread_start_us":326,"threads_started":1}
I20260925 01:29:15.357275 30551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.346001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:40803 (local address 127.0.0.1:58446)
0925 01:29:15.347245 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.347274 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.347403 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.349432 (+ 2029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.349439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.349480 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:15.350097 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.350115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.352211 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.352216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.354463 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.354474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.356326 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.356333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.356562 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.356598 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.356679 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.356821 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":34,"thread_start_us":1666,"threads_started":1}
I20260925 01:29:15.357610 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.346983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.27.160.2:40597)
0925 01:29:15.348598 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.348607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.348635 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:29:15.348748 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.348755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.348996 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.349628 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:15.350398 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.351762 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.354478 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.355912 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.356052 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.356284 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.356830 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.357044 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1434,"thread_start_us":1306,"threads_started":1}
I20260925 01:29:15.357842 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.346521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:40597 (local address 127.0.0.1:46932)
0925 01:29:15.347940 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.347961 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.348074 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.349247 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.349252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.349287 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:15.350201 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.350216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.351885 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.351893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.354286 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.354301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.356189 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.356198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.356390 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.356426 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.356537 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.356691 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"mutex_wait_us":212,"thread_start_us":387,"threads_started":1}
I20260925 01:29:15.358961 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.346543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.27.160.3:40803)
0925 01:29:15.348853 (+ 2310us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.348861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.348890 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:29:15.348974 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.348981 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.349236 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.349426 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:15.350329 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.352060 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.354627 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.356038 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.358145 (+ 2107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.358375 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.358440 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.358621 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2155,"thread_start_us":1970,"threads_started":1}
I20260925 01:29:15.361541 30326 tablet_service.cc:1467] Processing CreateTablet for tablet d6b04ba7fb42498a969a57cc256fcdbd (DEFAULT_TABLE table=test-table [id=9f2f0c3763dc44eb880ba7cd295761df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:15.361511 30186 tablet_service.cc:1467] Processing CreateTablet for tablet d6b04ba7fb42498a969a57cc256fcdbd (DEFAULT_TABLE table=test-table [id=9f2f0c3763dc44eb880ba7cd295761df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:15.362861 30326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b04ba7fb42498a969a57cc256fcdbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.363008 30186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b04ba7fb42498a969a57cc256fcdbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.364876 30465 tablet_service.cc:1467] Processing CreateTablet for tablet d6b04ba7fb42498a969a57cc256fcdbd (DEFAULT_TABLE table=test-table [id=9f2f0c3763dc44eb880ba7cd295761df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:15.366441 30465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6b04ba7fb42498a969a57cc256fcdbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.374177 30554 tablet_bootstrap.cc:492] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Bootstrap starting.
I20260925 01:29:15.378697 30554 tablet_bootstrap.cc:654] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:15.379070 30555 tablet_bootstrap.cc:492] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Bootstrap starting.
I20260925 01:29:15.380230 30554 log.cc:824] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:15.382833 30555 tablet_bootstrap.cc:654] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:15.383762 30557 tablet_bootstrap.cc:492] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Bootstrap starting.
I20260925 01:29:15.384070 30554 tablet_bootstrap.cc:492] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: No bootstrap required, opened a new log
I20260925 01:29:15.384146 30555 log.cc:824] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:15.384366 30554 ts_tablet_manager.cc:1397] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260925 01:29:15.387100 30557 tablet_bootstrap.cc:654] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:15.387298 30555 tablet_bootstrap.cc:492] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: No bootstrap required, opened a new log
I20260925 01:29:15.387576 30555 ts_tablet_manager.cc:1397] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:29:15.388280 30557 log.cc:824] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:15.391129 30557 tablet_bootstrap.cc:492] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: No bootstrap required, opened a new log
I20260925 01:29:15.391454 30557 ts_tablet_manager.cc:1397] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:29:15.394927 30554 raft_consensus.cc:348] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.395402 30554 raft_consensus.cc:374] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:15.395498 30554 raft_consensus.cc:729] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5781d682be4d96924dae671f6e68ce, State: Initialized, Role: FOLLOWER
I20260925 01:29:15.396137 30554 consensus_queue.cc:260] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.397352 30396 heartbeater.cc:503] Master 127.27.160.62:44805 was elected leader, sending a full tablet report...
I20260925 01:29:15.398142 30554 ts_tablet_manager.cc:1428] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260925 01:29:15.398144 30555 raft_consensus.cc:348] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.398604 30555 raft_consensus.cc:374] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:15.398703 30555 raft_consensus.cc:729] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013664afced241cd81da058a46acb28a, State: Initialized, Role: FOLLOWER
I20260925 01:29:15.399389 30555 consensus_queue.cc:260] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.401227 30555 ts_tablet_manager.cc:1428] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260925 01:29:15.403483 30257 tablet.cc:2369] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:15.404906 30557 raft_consensus.cc:348] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.405423 30557 raft_consensus.cc:374] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:15.405522 30557 raft_consensus.cc:729] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e04f723ffc84d19a9b773129d6d93a6, State: Initialized, Role: FOLLOWER
I20260925 01:29:15.406159 30557 consensus_queue.cc:260] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.407397 30535 heartbeater.cc:503] Master 127.27.160.62:44805 was elected leader, sending a full tablet report...
I20260925 01:29:15.407992 30557 ts_tablet_manager.cc:1428] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260925 01:29:15.432229 30560 raft_consensus.cc:482] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:15.432648 30560 raft_consensus.cc:504] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.434669 30560 leader_election.cc:288] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 013664afced241cd81da058a46acb28a (127.27.160.1:46579), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:15.441442 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.435095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:38571 (local address 127.27.160.1:46579)
0925 01:29:15.435329 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.435337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.436082 (+ 745us) server_negotiation.cc:407] Connection header received
0925 01:29:15.436326 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.436330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.436400 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.436499 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:15.437077 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.437844 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.439635 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.440495 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.440689 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.440763 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.441038 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.441182 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 01:29:15.441511 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.435098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.27.160.2:38571)
0925 01:29:15.435956 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.435976 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.436172 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.436515 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.436519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.436542 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:15.436923 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.436931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.437975 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.437983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.439449 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.439461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.440711 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.440717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.440879 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.440903 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.441011 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.441182 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":592,"threads_started":1}
I20260925 01:29:15.442503 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b04ba7fb42498a969a57cc256fcdbd" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013664afced241cd81da058a46acb28a" is_pre_election: true
I20260925 01:29:15.442883 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.435168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:57431 (local address 127.27.160.3:40803)
0925 01:29:15.435431 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.435440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.436144 (+ 704us) server_negotiation.cc:407] Connection header received
0925 01:29:15.436370 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.436375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.436443 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.436547 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:15.437414 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.438166 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.440124 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.441251 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.441431 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.441502 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.442470 (+ 968us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.442627 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 01:29:15.443110 30210 raft_consensus.cc:2454] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 0.
I20260925 01:29:15.443181 30564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.435159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:40803 (local address 127.27.160.2:57431)
0925 01:29:15.436024 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.436039 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.436205 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.436574 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.436579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.436610 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:15.437113 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.437144 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.438315 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.438323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.439962 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.439974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.442053 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.442060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.442277 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.442321 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.442423 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.442562 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":391,"threads_started":1}
I20260925 01:29:15.443894 30278 leader_election.cc:302] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013664afced241cd81da058a46acb28a, bb5781d682be4d96924dae671f6e68ce; no voters:
I20260925 01:29:15.444133 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b04ba7fb42498a969a57cc256fcdbd" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6" is_pre_election: true
I20260925 01:29:15.444350 30560 raft_consensus.cc:2781] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:15.444500 30560 raft_consensus.cc:482] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:15.444602 30560 raft_consensus.cc:3037] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:15.444700 30490 raft_consensus.cc:2454] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 0.
I20260925 01:29:15.448000 30560 raft_consensus.cc:504] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.448748 30560 leader_election.cc:288] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 election: Requested vote from peers 013664afced241cd81da058a46acb28a (127.27.160.1:46579), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:15.449199 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b04ba7fb42498a969a57cc256fcdbd" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6"
I20260925 01:29:15.449246 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6b04ba7fb42498a969a57cc256fcdbd" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013664afced241cd81da058a46acb28a"
I20260925 01:29:15.449410 30490 raft_consensus.cc:3037] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:15.449457 30210 raft_consensus.cc:3037] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:15.452066 30490 raft_consensus.cc:2454] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 1.
I20260925 01:29:15.452453 30276 leader_election.cc:302] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e04f723ffc84d19a9b773129d6d93a6, bb5781d682be4d96924dae671f6e68ce; no voters:
I20260925 01:29:15.452773 30560 raft_consensus.cc:2781] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:15.452888 30210 raft_consensus.cc:2454] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 1.
I20260925 01:29:15.453343 30560 raft_consensus.cc:686] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 1 LEADER]: Becoming Leader. State: Replica: bb5781d682be4d96924dae671f6e68ce, State: Running, Role: LEADER
I20260925 01:29:15.453886 30560 consensus_queue.cc:237] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.459239 30048 catalog_manager.cc:5495] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5781d682be4d96924dae671f6e68ce (127.27.160.2). New cstate: current_term: 1 leader_uuid: "bb5781d682be4d96924dae671f6e68ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
W20260925 01:29:15.464571 30397 tablet.cc:2369] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 01:29:15 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299753, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
W20260925 01:29:15.507719 30536 tablet.cc:2369] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:15.513418 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.499678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44805 (local address 127.0.0.1:44348)
0925 01:29:15.500280 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.500295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.500463 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.500769 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.500773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.501558 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:15.501813 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.501822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.502859 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.502867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.503798 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.503808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.504493 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.504498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.504658 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.505331 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:15.505346 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:15.507698 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:15.510949 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:15.510960 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:15.510965 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:15.511383 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:15.511803 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:15.511806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:15.511809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:15.511976 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:15.512518 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:15.512523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:15.512676 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.512814 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.513010 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":316,"threads_started":1}
I20260925 01:29:15.513943 30284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.499647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44348 (local address 127.27.160.62:44805)
0925 01:29:15.499866 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.499871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.500385 (+ 514us) server_negotiation.cc:407] Connection header received
0925 01:29:15.500597 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.500600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.500651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.500747 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:15.501972 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.502733 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.503933 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.504319 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.504453 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.507927 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:15.507933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:15.507936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:15.507964 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:15.510803 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:15.511534 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:15.511539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:15.511541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:15.511694 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:15.512099 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:15.512103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:15.512105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:15.512395 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:15.512550 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.513024 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.513299 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:29:15.519618 30048 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:44348:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:29:15.521070 30048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:15.528362 30186 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd937c5b8841b98117cb98d67225d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=442d13a2a53f4ff88e4bfd16d039645e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:29:15.528805 30326 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd937c5b8841b98117cb98d67225d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=442d13a2a53f4ff88e4bfd16d039645e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:29:15.528797 30465 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd937c5b8841b98117cb98d67225d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=442d13a2a53f4ff88e4bfd16d039645e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:29:15.529491 30186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd937c5b8841b98117cb98d67225d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.529672 30465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd937c5b8841b98117cb98d67225d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.529703 30326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd937c5b8841b98117cb98d67225d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.537701 30555 tablet_bootstrap.cc:492] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: Bootstrap starting.
I20260925 01:29:15.537886 30554 tablet_bootstrap.cc:492] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: Bootstrap starting.
I20260925 01:29:15.539292 30554 tablet_bootstrap.cc:654] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:15.539570 30555 tablet_bootstrap.cc:654] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:15.541183 30557 tablet_bootstrap.cc:492] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: Bootstrap starting.
I20260925 01:29:15.542254 30554 tablet_bootstrap.cc:492] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: No bootstrap required, opened a new log
I20260925 01:29:15.542423 30554 ts_tablet_manager.cc:1397] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 01:29:15.542622 30557 tablet_bootstrap.cc:654] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:15.543148 30554 raft_consensus.cc:348] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.543366 30554 raft_consensus.cc:374] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:15.543473 30554 raft_consensus.cc:729] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5781d682be4d96924dae671f6e68ce, State: Initialized, Role: FOLLOWER
I20260925 01:29:15.543731 30554 consensus_queue.cc:260] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.544340 30554 ts_tablet_manager.cc:1428] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 01:29:15.544162 30560 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.544528 30560 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:15.544667 30557 tablet_bootstrap.cc:492] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: No bootstrap required, opened a new log
I20260925 01:29:15.544814 30557 ts_tablet_manager.cc:1397] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 01:29:15.545564 30557 raft_consensus.cc:348] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.545683 30555 tablet_bootstrap.cc:492] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: No bootstrap required, opened a new log
I20260925 01:29:15.545743 30557 raft_consensus.cc:374] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:15.545861 30557 raft_consensus.cc:729] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e04f723ffc84d19a9b773129d6d93a6, State: Initialized, Role: FOLLOWER
I20260925 01:29:15.545979 30555 ts_tablet_manager.cc:1397] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260925 01:29:15.546116 30557 consensus_queue.cc:260] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.546578 30562 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.546921 30562 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:15.547066 30555 raft_consensus.cc:348] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.547468 30555 raft_consensus.cc:374] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:15.547622 30555 raft_consensus.cc:729] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013664afced241cd81da058a46acb28a, State: Initialized, Role: FOLLOWER
I20260925 01:29:15.547775 30557 ts_tablet_manager.cc:1428] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260925 01:29:15.548756 30555 consensus_queue.cc:260] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.549463 30555 ts_tablet_manager.cc:1428] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:29:15.549281 30561 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.549705 30561 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:15.705772 30561 raft_consensus.cc:482] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:15.706033 30561 raft_consensus.cc:504] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.707561 30561 leader_election.cc:288] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5781d682be4d96924dae671f6e68ce (127.27.160.2:40597), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:15.712785 30560 raft_consensus.cc:482] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:15.712996 30560 raft_consensus.cc:504] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.713745 30560 leader_election.cc:288] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 013664afced241cd81da058a46acb28a (127.27.160.1:46579), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:15.714313 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd937c5b8841b98117cb98d67225d1" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013664afced241cd81da058a46acb28a" is_pre_election: true
I20260925 01:29:15.714387 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd937c5b8841b98117cb98d67225d1" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6" is_pre_election: true
I20260925 01:29:15.714577 30210 raft_consensus.cc:2454] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 0.
I20260925 01:29:15.714782 30490 raft_consensus.cc:2454] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 0.
I20260925 01:29:15.715010 30278 leader_election.cc:302] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013664afced241cd81da058a46acb28a, bb5781d682be4d96924dae671f6e68ce; no voters:
I20260925 01:29:15.715760 30560 raft_consensus.cc:2781] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:15.715867 30560 raft_consensus.cc:482] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:15.715996 30560 raft_consensus.cc:3037] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:15.717386 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.708070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:43899 (local address 127.27.160.3:40803)
0925 01:29:15.708290 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.708297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.709171 (+ 874us) server_negotiation.cc:407] Connection header received
0925 01:29:15.709405 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.709409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.709469 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.709568 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:15.710215 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.711426 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.714268 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.715918 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.716050 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.716120 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.716863 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.717069 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:29:15.717691 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.708140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:57969 (local address 127.27.160.2:40597)
0925 01:29:15.708364 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.708372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.709154 (+ 782us) server_negotiation.cc:407] Connection header received
0925 01:29:15.709413 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.709417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.709483 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.709578 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:15.710335 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.711478 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.714253 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.715586 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.715736 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.715824 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.716636 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.716783 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:29:15.717790 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.708118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:40803 (local address 127.27.160.1:43899)
0925 01:29:15.709012 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.709028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.709250 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.709589 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.709593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.709614 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:15.710029 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.710041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.711601 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.711610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.714092 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.714107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.716419 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.716440 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.716652 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.716692 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.717118 (+ 426us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.717294 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":4,"spinlock_wait_cycles":7552,"thread_start_us":578,"threads_started":1}
I20260925 01:29:15.718113 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.707985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:40597 (local address 127.27.160.1:57969)
0925 01:29:15.709010 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.709027 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.709260 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.709606 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.709610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.709633 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:15.710095 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.710106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.711619 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.711624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.714049 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.714078 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.716186 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.716193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.716395 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.716442 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.717182 (+ 740us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.717320 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":154,"spinlock_wait_cycles":4352,"thread_start_us":618,"threads_started":1}
I20260925 01:29:15.718572 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd937c5b8841b98117cb98d67225d1" candidate_uuid: "013664afced241cd81da058a46acb28a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6" is_pre_election: true
I20260925 01:29:15.718869 30490 raft_consensus.cc:2454] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 013664afced241cd81da058a46acb28a in term 0.
I20260925 01:29:15.719077 30351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd937c5b8841b98117cb98d67225d1" candidate_uuid: "013664afced241cd81da058a46acb28a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5781d682be4d96924dae671f6e68ce" is_pre_election: true
I20260925 01:29:15.719462 30136 leader_election.cc:302] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013664afced241cd81da058a46acb28a, 7e04f723ffc84d19a9b773129d6d93a6; no voters:
I20260925 01:29:15.719893 30561 raft_consensus.cc:2781] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:15.720019 30561 raft_consensus.cc:482] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:15.720093 30561 raft_consensus.cc:3037] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:15.721153 30560 raft_consensus.cc:504] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.721719 30351 raft_consensus.cc:2379] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 013664afced241cd81da058a46acb28a in current term 1: Already voted for candidate bb5781d682be4d96924dae671f6e68ce in this term.
I20260925 01:29:15.722455 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd937c5b8841b98117cb98d67225d1" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013664afced241cd81da058a46acb28a"
I20260925 01:29:15.723085 30561 raft_consensus.cc:504] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.723742 30210 raft_consensus.cc:2379] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bb5781d682be4d96924dae671f6e68ce in current term 1: Already voted for candidate 013664afced241cd81da058a46acb28a in this term.
I20260925 01:29:15.723806 30561 leader_election.cc:288] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [CANDIDATE]: Term 1 election: Requested vote from peers bb5781d682be4d96924dae671f6e68ce (127.27.160.2:40597), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:15.724388 30351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd937c5b8841b98117cb98d67225d1" candidate_uuid: "013664afced241cd81da058a46acb28a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5781d682be4d96924dae671f6e68ce"
I20260925 01:29:15.724679 30351 raft_consensus.cc:2379] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 013664afced241cd81da058a46acb28a in current term 1: Already voted for candidate bb5781d682be4d96924dae671f6e68ce in this term.
I20260925 01:29:15.725068 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd937c5b8841b98117cb98d67225d1" candidate_uuid: "013664afced241cd81da058a46acb28a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6"
I20260925 01:29:15.725365 30490 raft_consensus.cc:3037] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:15.727437 30490 raft_consensus.cc:2454] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 013664afced241cd81da058a46acb28a in term 1.
I20260925 01:29:15.727885 30560 leader_election.cc:288] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 election: Requested vote from peers 013664afced241cd81da058a46acb28a (127.27.160.1:46579), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:15.727896 30136 leader_election.cc:302] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 013664afced241cd81da058a46acb28a, 7e04f723ffc84d19a9b773129d6d93a6; no voters: bb5781d682be4d96924dae671f6e68ce
I20260925 01:29:15.728255 30561 raft_consensus.cc:2781] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:15.728436 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd937c5b8841b98117cb98d67225d1" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6"
I20260925 01:29:15.728727 30490 raft_consensus.cc:2379] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bb5781d682be4d96924dae671f6e68ce in current term 1: Already voted for candidate 013664afced241cd81da058a46acb28a in this term.
I20260925 01:29:15.729331 30561 raft_consensus.cc:686] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [term 1 LEADER]: Becoming Leader. State: Replica: 013664afced241cd81da058a46acb28a, State: Running, Role: LEADER
I20260925 01:29:15.729827 30276 leader_election.cc:302] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb5781d682be4d96924dae671f6e68ce; no voters: 013664afced241cd81da058a46acb28a, 7e04f723ffc84d19a9b773129d6d93a6
I20260925 01:29:15.729966 30561 consensus_queue.cc:237] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:15.730437 30560 raft_consensus.cc:2726] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 01:29:15.733053 30578 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: TxnStatusTablet state changed. Reason: New leader 013664afced241cd81da058a46acb28a. Latest consensus state: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.733538 30578 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:29:15.734381 30580 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:29:15.734694 30580 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:29:15.735311 30048 catalog_manager.cc:5495] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a reported cstate change: term changed from 0 to 1, leader changed from <none> to 013664afced241cd81da058a46acb28a (127.27.160.1). New cstate: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260925 01:29:15.791769 30572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.785373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.0.0.1:39900)
0925 01:29:15.785632 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.785650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.785763 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.786224 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.786227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.786237 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:29:15.786394 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.786401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.787870 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.787874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.789031 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.789047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.790082 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.790089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.790265 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.790322 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:29:15.791325 (+ 1003us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:29:15.791343 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.791401 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.791543 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 01:29:15.791842 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.785386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.27.160.1:46579)
0925 01:29:15.785747 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.785753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.785775 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:29:15.785933 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.785939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.786037 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.786206 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:15.786525 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.787734 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.789238 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.789890 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.790068 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.790503 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:15.791217 (+ 714us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:15.791376 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.791424 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.791515 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260925 01:29:15.800601 30490 raft_consensus.cc:1261] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Refusing update from remote peer 013664afced241cd81da058a46acb28a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:15.800642 30351 raft_consensus.cc:1261] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Refusing update from remote peer 013664afced241cd81da058a46acb28a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:15.801498 30578 consensus_queue.cc:1035] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:15.801859 30561 consensus_queue.cc:1035] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:15.812017 30562 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: TxnStatusTablet state changed. Reason: New leader 013664afced241cd81da058a46acb28a. Latest consensus state: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.812325 30562 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:15.814774 30583 mvcc.cc:205] Tried to move back new op lower bound from 7333067799749734400 to 7333067799474610176. Current Snapshot: MvccSnapshot[applied={T|T < 7333067799749734400}]
I20260925 01:29:15.823326 30578 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.823736 30578 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:29:15.827104 30562 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.827463 30562 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P 7e04f723ffc84d19a9b773129d6d93a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:15.828377 30560 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: TxnStatusTablet state changed. Reason: New leader 013664afced241cd81da058a46acb28a. Latest consensus state: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.828680 30560 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:15.832183 30578 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.832454 30578 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P 013664afced241cd81da058a46acb28a: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:29:15.836077 30560 tablet_replica.cc:439] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } } }
I20260925 01:29:15.836357 30560 tablet_replica.cc:442] T 66cd937c5b8841b98117cb98d67225d1 P bb5781d682be4d96924dae671f6e68ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:29:15.845067 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.838039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.27.160.2:40597)
0925 01:29:15.838210 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.838218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.838911 (+ 693us) server_negotiation.cc:407] Connection header received
0925 01:29:15.839217 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.839222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.839314 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.839465 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:15.839895 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.841042 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.842474 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.843122 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.843251 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.843676 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:15.844353 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:15.844505 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.844657 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.844743 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:29:15.863734 30549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.856914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:60893 (local address 127.27.160.1:46579)
0925 01:29:15.857215 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.857223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.857836 (+ 613us) server_negotiation.cc:407] Connection header received
0925 01:29:15.858135 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.858143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.858255 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.858348 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:15.858928 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.859842 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.861881 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.862324 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.862422 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.863060 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:15.863185 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:15.863265 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.863427 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.863498 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:29:15.863929 30595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.856896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.27.160.2:60893)
0925 01:29:15.857710 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.857734 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.857938 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.858394 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.858399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.858412 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:29:15.858758 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.858769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.859985 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.859993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.861736 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.861752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.862503 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.862510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.862701 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.862778 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:29:15.863291 (+ 513us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:29:15.863303 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.863389 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.863542 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":423,"threads_started":1}
I20260925 01:29:15.873643 30490 raft_consensus.cc:1261] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Refusing update from remote peer bb5781d682be4d96924dae671f6e68ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:15.874231 30210 raft_consensus.cc:1261] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Refusing update from remote peer bb5781d682be4d96924dae671f6e68ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:15.874296 30560 consensus_queue.cc:1035] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:15.874888 30565 consensus_queue.cc:1035] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:15.883550 30585 mvcc.cc:205] Tried to move back new op lower bound from 7333067800052621312 to 7333067798344060928. Current Snapshot: MvccSnapshot[applied={T|T < 7333067800052621312}]
I20260925 01:29:15.884088 30584 mvcc.cc:205] Tried to move back new op lower bound from 7333067800052621312 to 7333067798344060928. Current Snapshot: MvccSnapshot[applied={T|T < 7333067800052621312}]
I20260925 01:29:15.919486 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.907579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:45095 (local address 127.27.160.2:40597)
0925 01:29:15.907782 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:15.907789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:15.913168 (+ 5379us) server_negotiation.cc:407] Connection header received
0925 01:29:15.913429 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:15.913435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:15.913533 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:15.913610 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:15.914285 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.915235 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.917060 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.917595 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.917730 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.918760 (+ 1030us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:15.918926 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:15.919043 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:15.919175 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 01:29:15.919248 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:29:15.919725 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:15.907555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:40597 (local address 127.27.160.1:45095)
0925 01:29:15.913033 (+ 5478us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:15.913066 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:15.913256 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:15.913662 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:15.913667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:15.913687 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:29:15.914113 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.914126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.915388 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.915393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.916930 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:15.916947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:15.917794 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:15.917801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:15.917991 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:15.918408 (+ 417us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:29:15.919051 (+ 643us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:29:15.919071 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 01:29:15.919162 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 01:29:15.919309 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5237,"thread_start_us":5079,"threads_started":1}
I20260925 01:29:15.966559 30048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:15.966857 30048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:15.970865 30048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: Sending DeleteTablet for 3 replicas of tablet d6b04ba7fb42498a969a57cc256fcdbd
I20260925 01:29:15.972173 30465 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b04ba7fb42498a969a57cc256fcdbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:15 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:58446
I20260925 01:29:15.974471 30048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:15.975142 30048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:15.976293 30326 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b04ba7fb42498a969a57cc256fcdbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:15 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:46932
I20260925 01:29:15.977290 30186 tablet_service.cc:1514] Processing DeleteTablet for tablet d6b04ba7fb42498a969a57cc256fcdbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:15 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:39892
I20260925 01:29:15.985219 30464 tablet_service.cc:1467] Processing CreateTablet for tablet 603212a761d745d19571b0d73b5ae2e3 (DEFAULT_TABLE table=test-table [id=8a7d540079a34fbabb76af4bb93f5dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:15.985705 30464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603212a761d745d19571b0d73b5ae2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.990203 30325 tablet_service.cc:1467] Processing CreateTablet for tablet 603212a761d745d19571b0d73b5ae2e3 (DEFAULT_TABLE table=test-table [id=8a7d540079a34fbabb76af4bb93f5dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:15.990751 30325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603212a761d745d19571b0d73b5ae2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.993206 30606 tablet_replica.cc:330] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: stopping tablet replica
I20260925 01:29:15.994690 30606 raft_consensus.cc:2229] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:15.994807 30185 tablet_service.cc:1467] Processing CreateTablet for tablet 603212a761d745d19571b0d73b5ae2e3 (DEFAULT_TABLE table=test-table [id=8a7d540079a34fbabb76af4bb93f5dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:15.995086 30606 raft_consensus.cc:2258] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:15.995442 30185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603212a761d745d19571b0d73b5ae2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:15.999202 30606 ts_tablet_manager.cc:1905] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:16.013224 30606 ts_tablet_manager.cc:1918] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:16.013418 30606 log.cc:1196] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/d6b04ba7fb42498a969a57cc256fcdbd
I20260925 01:29:16.013978 30606 ts_tablet_manager.cc:1939] T d6b04ba7fb42498a969a57cc256fcdbd P 013664afced241cd81da058a46acb28a: Deleting consensus metadata
I20260925 01:29:16.015694 30557 tablet_bootstrap.cc:492] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: Bootstrap starting.
I20260925 01:29:16.017725 30557 tablet_bootstrap.cc:654] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:16.018952 30554 tablet_bootstrap.cc:492] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: Bootstrap starting.
I20260925 01:29:16.020756 30554 tablet_bootstrap.cc:654] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:16.021387 30034 catalog_manager.cc:4841] TS 013664afced241cd81da058a46acb28a (127.27.160.1:46579): tablet d6b04ba7fb42498a969a57cc256fcdbd (table test-table [id=9f2f0c3763dc44eb880ba7cd295761df]) successfully deleted
I20260925 01:29:16.022078 30555 tablet_bootstrap.cc:492] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: Bootstrap starting.
I20260925 01:29:16.024195 30555 tablet_bootstrap.cc:654] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:16.024745 30605 tablet_replica.cc:330] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: stopping tablet replica
I20260925 01:29:16.025785 30605 raft_consensus.cc:2229] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:16.026575 30605 raft_consensus.cc:2258] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:16.028697 30605 ts_tablet_manager.cc:1905] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:16.037406 30605 ts_tablet_manager.cc:1918] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:16.038117 30604 tablet_replica.cc:330] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: stopping tablet replica
I20260925 01:29:16.038681 30604 raft_consensus.cc:2229] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:16.039192 30604 raft_consensus.cc:2258] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:16.041352 30604 ts_tablet_manager.cc:1905] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:16.039660 30605 log.cc:1196] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/d6b04ba7fb42498a969a57cc256fcdbd
I20260925 01:29:16.044973 30605 ts_tablet_manager.cc:1939] T d6b04ba7fb42498a969a57cc256fcdbd P bb5781d682be4d96924dae671f6e68ce: Deleting consensus metadata
I20260925 01:29:16.048385 30034 catalog_manager.cc:4841] TS bb5781d682be4d96924dae671f6e68ce (127.27.160.2:40597): tablet d6b04ba7fb42498a969a57cc256fcdbd (table test-table [id=9f2f0c3763dc44eb880ba7cd295761df]) successfully deleted
I20260925 01:29:16.049372 30604 ts_tablet_manager.cc:1918] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:16.049563 30604 log.cc:1196] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/d6b04ba7fb42498a969a57cc256fcdbd
I20260925 01:29:16.050119 30604 ts_tablet_manager.cc:1939] T d6b04ba7fb42498a969a57cc256fcdbd P 7e04f723ffc84d19a9b773129d6d93a6: Deleting consensus metadata
I20260925 01:29:16.052007 30032 catalog_manager.cc:4841] TS 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803): tablet d6b04ba7fb42498a969a57cc256fcdbd (table test-table [id=9f2f0c3763dc44eb880ba7cd295761df]) successfully deleted
I20260925 01:29:16.053879 30555 tablet_bootstrap.cc:492] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: No bootstrap required, opened a new log
I20260925 01:29:16.054081 30555 ts_tablet_manager.cc:1397] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.000s
I20260925 01:29:16.054935 30555 raft_consensus.cc:348] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.055230 30555 raft_consensus.cc:374] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:16.055330 30555 raft_consensus.cc:729] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013664afced241cd81da058a46acb28a, State: Initialized, Role: FOLLOWER
I20260925 01:29:16.056329 30555 consensus_queue.cc:260] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.057343 30555 ts_tablet_manager.cc:1428] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:16.059566 30554 tablet_bootstrap.cc:492] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: No bootstrap required, opened a new log
I20260925 01:29:16.059806 30554 ts_tablet_manager.cc:1397] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.013s
I20260925 01:29:16.060742 30554 raft_consensus.cc:348] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.061101 30554 raft_consensus.cc:374] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:16.061266 30554 raft_consensus.cc:729] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5781d682be4d96924dae671f6e68ce, State: Initialized, Role: FOLLOWER
I20260925 01:29:16.061605 30554 consensus_queue.cc:260] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.062604 30554 ts_tablet_manager.cc:1428] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 01:29:16.063493 30557 tablet_bootstrap.cc:492] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: No bootstrap required, opened a new log
I20260925 01:29:16.063732 30557 ts_tablet_manager.cc:1397] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.004s
I20260925 01:29:16.064571 30557 raft_consensus.cc:348] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.064888 30557 raft_consensus.cc:374] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:16.065011 30557 raft_consensus.cc:729] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e04f723ffc84d19a9b773129d6d93a6, State: Initialized, Role: FOLLOWER
I20260925 01:29:16.065374 30557 consensus_queue.cc:260] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.066010 30557 ts_tablet_manager.cc:1428] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 01:29:16.069645 30601 raft_consensus.cc:482] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:16.069839 30601 raft_consensus.cc:504] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.070677 30601 leader_election.cc:288] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 013664afced241cd81da058a46acb28a (127.27.160.1:46579), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:16.071529 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603212a761d745d19571b0d73b5ae2e3" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013664afced241cd81da058a46acb28a" is_pre_election: true
I20260925 01:29:16.071780 30210 raft_consensus.cc:2454] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 0.
I20260925 01:29:16.072291 30278 leader_election.cc:302] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013664afced241cd81da058a46acb28a, bb5781d682be4d96924dae671f6e68ce; no voters:
I20260925 01:29:16.072379 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603212a761d745d19571b0d73b5ae2e3" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6" is_pre_election: true
I20260925 01:29:16.072590 30490 raft_consensus.cc:2454] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 0.
I20260925 01:29:16.072873 30601 raft_consensus.cc:2781] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:16.073004 30601 raft_consensus.cc:482] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:16.073108 30601 raft_consensus.cc:3037] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:16.076875 30601 raft_consensus.cc:504] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.078043 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603212a761d745d19571b0d73b5ae2e3" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6"
I20260925 01:29:16.078299 30490 raft_consensus.cc:3037] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:16.081324 30490 raft_consensus.cc:2454] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 1.
I20260925 01:29:16.081405 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603212a761d745d19571b0d73b5ae2e3" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013664afced241cd81da058a46acb28a"
I20260925 01:29:16.081640 30210 raft_consensus.cc:3037] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:16.081881 30276 leader_election.cc:302] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e04f723ffc84d19a9b773129d6d93a6, bb5781d682be4d96924dae671f6e68ce; no voters:
I20260925 01:29:16.082190 30565 raft_consensus.cc:2781] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:16.082356 30565 raft_consensus.cc:686] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 1 LEADER]: Becoming Leader. State: Replica: bb5781d682be4d96924dae671f6e68ce, State: Running, Role: LEADER
I20260925 01:29:16.082594 30565 consensus_queue.cc:237] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:16.085250 30210 raft_consensus.cc:2454] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 1.
I20260925 01:29:16.085759 30601 leader_election.cc:288] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 election: Requested vote from peers 013664afced241cd81da058a46acb28a (127.27.160.1:46579), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:16.088142 30048 catalog_manager.cc:5495] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5781d682be4d96924dae671f6e68ce (127.27.160.2). New cstate: current_term: 1 leader_uuid: "bb5781d682be4d96924dae671f6e68ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260925 01:29:16.121249 30489 raft_consensus.cc:1261] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Refusing update from remote peer bb5781d682be4d96924dae671f6e68ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:16.121376 30210 raft_consensus.cc:1261] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Refusing update from remote peer bb5781d682be4d96924dae671f6e68ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:16.121865 30565 consensus_queue.cc:1035] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:16.122226 30560 consensus_queue.cc:1035] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:29:16.170207 30585 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 01:29:17.780714 30046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:17.780927 30046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:17.785480 30046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: Sending DeleteTablet for 3 replicas of tablet 603212a761d745d19571b0d73b5ae2e3
I20260925 01:29:17.786423 30465 tablet_service.cc:1514] Processing DeleteTablet for tablet 603212a761d745d19571b0d73b5ae2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:17 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:58446
I20260925 01:29:17.786484 30326 tablet_service.cc:1514] Processing DeleteTablet for tablet 603212a761d745d19571b0d73b5ae2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:17 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:46932
I20260925 01:29:17.786569 30185 tablet_service.cc:1514] Processing DeleteTablet for tablet 603212a761d745d19571b0d73b5ae2e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:17 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:39892
I20260925 01:29:17.798875 30636 tablet_replica.cc:330] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: stopping tablet replica
I20260925 01:29:17.799131 30636 raft_consensus.cc:2229] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:17.799223 30638 tablet_replica.cc:330] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: stopping tablet replica
I20260925 01:29:17.799398 30636 raft_consensus.cc:2258] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:17.799463 30637 tablet_replica.cc:330] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: stopping tablet replica
I20260925 01:29:17.799453 30638 raft_consensus.cc:2229] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:17.799650 30637 raft_consensus.cc:2229] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:17.799710 30638 raft_consensus.cc:2258] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:17.800040 30637 raft_consensus.cc:2258] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:17.800613 30636 ts_tablet_manager.cc:1905] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:17.800765 30638 ts_tablet_manager.cc:1905] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:17.801059 30637 ts_tablet_manager.cc:1905] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:17.805954 30638 ts_tablet_manager.cc:1918] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:29:17.806103 30638 log.cc:1196] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/603212a761d745d19571b0d73b5ae2e3
I20260925 01:29:17.806612 30638 ts_tablet_manager.cc:1939] T 603212a761d745d19571b0d73b5ae2e3 P 013664afced241cd81da058a46acb28a: Deleting consensus metadata
I20260925 01:29:17.807657 30637 ts_tablet_manager.cc:1918] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:29:17.807852 30637 log.cc:1196] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/603212a761d745d19571b0d73b5ae2e3
I20260925 01:29:17.807934 30034 catalog_manager.cc:4841] TS 013664afced241cd81da058a46acb28a (127.27.160.1:46579): tablet 603212a761d745d19571b0d73b5ae2e3 (table test-table [id=8a7d540079a34fbabb76af4bb93f5dbb]) successfully deleted
I20260925 01:29:17.808039 30636 ts_tablet_manager.cc:1918] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:29:17.808275 30637 ts_tablet_manager.cc:1939] T 603212a761d745d19571b0d73b5ae2e3 P bb5781d682be4d96924dae671f6e68ce: Deleting consensus metadata
I20260925 01:29:17.808339 30636 log.cc:1196] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/603212a761d745d19571b0d73b5ae2e3
I20260925 01:29:17.808743 30636 ts_tablet_manager.cc:1939] T 603212a761d745d19571b0d73b5ae2e3 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting consensus metadata
I20260925 01:29:17.809434 30034 catalog_manager.cc:4841] TS bb5781d682be4d96924dae671f6e68ce (127.27.160.2:40597): tablet 603212a761d745d19571b0d73b5ae2e3 (table test-table [id=8a7d540079a34fbabb76af4bb93f5dbb]) successfully deleted
I20260925 01:29:17.809806 30032 catalog_manager.cc:4841] TS 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803): tablet 603212a761d745d19571b0d73b5ae2e3 (table test-table [id=8a7d540079a34fbabb76af4bb93f5dbb]) successfully deleted
Sep 25 01:29:17 dist-test-slave-h6hf krb5kdc[30003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:29:17 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:17.847967 30646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:17.833813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.27.160.62:44805)
0925 01:29:17.834409 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:17.834415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:17.834467 (+ 52us) server_negotiation.cc:407] Connection header received
0925 01:29:17.834526 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:17.834530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:17.834589 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:17.834675 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:17.836022 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:17.836837 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:17.838131 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:17.838513 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:17.838631 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:17.842658 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:17.842664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:17.842667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:17.842703 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:17.845539 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:17.846165 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:17.846169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:17.846172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:17.846292 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:17.846584 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:17.846587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:17.846589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:17.846853 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:17.847048 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:17.847372 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 01:29:17.847553 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":343,"threads_started":1}
I20260925 01:29:17.851466 30046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:17.852139 30046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:17.859176 30185 tablet_service.cc:1467] Processing CreateTablet for tablet 94f02f8148f248e1adab00209e34fae6 (DEFAULT_TABLE table=test-table [id=6a4e23b6038b431392ace720941924ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:17.859197 30326 tablet_service.cc:1467] Processing CreateTablet for tablet 94f02f8148f248e1adab00209e34fae6 (DEFAULT_TABLE table=test-table [id=6a4e23b6038b431392ace720941924ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:17.859714 30185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f02f8148f248e1adab00209e34fae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:17.859699 30326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f02f8148f248e1adab00209e34fae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:17.859651 30465 tablet_service.cc:1467] Processing CreateTablet for tablet 94f02f8148f248e1adab00209e34fae6 (DEFAULT_TABLE table=test-table [id=6a4e23b6038b431392ace720941924ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:17.860056 30465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f02f8148f248e1adab00209e34fae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:17.866027 30647 tablet_bootstrap.cc:492] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: Bootstrap starting.
I20260925 01:29:17.867379 30648 tablet_bootstrap.cc:492] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: Bootstrap starting.
I20260925 01:29:17.867864 30647 tablet_bootstrap.cc:654] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:17.868312 30649 tablet_bootstrap.cc:492] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: Bootstrap starting.
I20260925 01:29:17.869052 30648 tablet_bootstrap.cc:654] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:17.870390 30649 tablet_bootstrap.cc:654] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:17.871050 30647 tablet_bootstrap.cc:492] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: No bootstrap required, opened a new log
I20260925 01:29:17.871173 30648 tablet_bootstrap.cc:492] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: No bootstrap required, opened a new log
I20260925 01:29:17.871210 30647 ts_tablet_manager.cc:1397] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 01:29:17.871309 30648 ts_tablet_manager.cc:1397] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:29:17.871999 30647 raft_consensus.cc:348] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:17.872012 30648 raft_consensus.cc:348] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:17.872248 30647 raft_consensus.cc:374] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:17.872262 30648 raft_consensus.cc:374] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:17.872341 30647 raft_consensus.cc:729] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e04f723ffc84d19a9b773129d6d93a6, State: Initialized, Role: FOLLOWER
I20260925 01:29:17.872346 30648 raft_consensus.cc:729] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013664afced241cd81da058a46acb28a, State: Initialized, Role: FOLLOWER
I20260925 01:29:17.872678 30649 tablet_bootstrap.cc:492] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: No bootstrap required, opened a new log
I20260925 01:29:17.872570 30647 consensus_queue.cc:260] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:17.872604 30648 consensus_queue.cc:260] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:17.872819 30649 ts_tablet_manager.cc:1397] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 01:29:17.873759 30649 raft_consensus.cc:348] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:17.873960 30648 ts_tablet_manager.cc:1428] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:29:17.874010 30649 raft_consensus.cc:374] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:17.874109 30649 raft_consensus.cc:729] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5781d682be4d96924dae671f6e68ce, State: Initialized, Role: FOLLOWER
I20260925 01:29:17.874466 30647 ts_tablet_manager.cc:1428] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:17.874397 30649 consensus_queue.cc:260] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:17.874992 30649 ts_tablet_manager.cc:1428] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 01:29:18.059031 30613 raft_consensus.cc:482] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:18.059247 30613 raft_consensus.cc:504] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.060068 30613 leader_election.cc:288] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb5781d682be4d96924dae671f6e68ce (127.27.160.2:40597), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:18.060678 30351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f02f8148f248e1adab00209e34fae6" candidate_uuid: "013664afced241cd81da058a46acb28a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5781d682be4d96924dae671f6e68ce" is_pre_election: true
I20260925 01:29:18.060678 30488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f02f8148f248e1adab00209e34fae6" candidate_uuid: "013664afced241cd81da058a46acb28a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6" is_pre_election: true
I20260925 01:29:18.060994 30351 raft_consensus.cc:2454] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 013664afced241cd81da058a46acb28a in term 0.
I20260925 01:29:18.061021 30488 raft_consensus.cc:2454] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 013664afced241cd81da058a46acb28a in term 0.
I20260925 01:29:18.061573 30138 leader_election.cc:302] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013664afced241cd81da058a46acb28a, bb5781d682be4d96924dae671f6e68ce; no voters:
I20260925 01:29:18.061872 30613 raft_consensus.cc:2781] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:18.062013 30613 raft_consensus.cc:482] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:18.062134 30613 raft_consensus.cc:3037] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:18.064201 30613 raft_consensus.cc:504] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.064801 30613 leader_election.cc:288] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [CANDIDATE]: Term 1 election: Requested vote from peers bb5781d682be4d96924dae671f6e68ce (127.27.160.2:40597), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:18.065310 30351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f02f8148f248e1adab00209e34fae6" candidate_uuid: "013664afced241cd81da058a46acb28a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb5781d682be4d96924dae671f6e68ce"
I20260925 01:29:18.065327 30488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f02f8148f248e1adab00209e34fae6" candidate_uuid: "013664afced241cd81da058a46acb28a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6"
I20260925 01:29:18.065531 30488 raft_consensus.cc:3037] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:18.065526 30351 raft_consensus.cc:3037] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:18.067551 30351 raft_consensus.cc:2454] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 013664afced241cd81da058a46acb28a in term 1.
I20260925 01:29:18.067570 30488 raft_consensus.cc:2454] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 013664afced241cd81da058a46acb28a in term 1.
I20260925 01:29:18.068012 30136 leader_election.cc:302] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013664afced241cd81da058a46acb28a, 7e04f723ffc84d19a9b773129d6d93a6; no voters:
I20260925 01:29:18.068326 30613 raft_consensus.cc:2781] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:18.068495 30613 raft_consensus.cc:686] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 1 LEADER]: Becoming Leader. State: Replica: 013664afced241cd81da058a46acb28a, State: Running, Role: LEADER
I20260925 01:29:18.068785 30613 consensus_queue.cc:237] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.071619 30046 catalog_manager.cc:5495] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a reported cstate change: term changed from 0 to 1, leader changed from <none> to 013664afced241cd81da058a46acb28a (127.27.160.1). New cstate: current_term: 1 leader_uuid: "013664afced241cd81da058a46acb28a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260925 01:29:18.118862 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:18.111815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.27.160.1:46579)
0925 01:29:18.113592 (+ 1777us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:18.113597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:18.113618 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:29:18.113691 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:18.113695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:18.113763 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:18.113898 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:18.114382 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:18.115191 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:18.116845 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:18.117325 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:18.117487 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:18.117996 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:18.118119 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:18.118266 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:18.118415 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 01:29:18.118506 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1662,"thread_start_us":1511,"threads_started":1}
I20260925 01:29:18.132808 30656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:18.127445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:40629 (local address 127.27.160.1:46579)
0925 01:29:18.127622 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:18.127626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:18.127998 (+ 372us) server_negotiation.cc:407] Connection header received
0925 01:29:18.128114 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:18.128117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:18.128176 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:18.128266 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:18.128808 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:18.129712 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:18.131167 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:18.131642 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:18.131767 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:18.132171 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:18.132271 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:18.132351 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:18.132475 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 01:29:18.132560 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":67}
I20260925 01:29:18.132861 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:18.127322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.27.160.1:40629)
0925 01:29:18.127890 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:18.127906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:18.128014 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:18.128306 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:18.128318 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:18.128333 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:29:18.128637 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:18.128653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:18.129827 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:18.129830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:18.131030 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:18.131042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:18.131830 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:18.131834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:18.131948 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:18.131983 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:29:18.132363 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:29:18.132372 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:29:18.132433 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 01:29:18.132545 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":1,"thread_start_us":330,"threads_started":1}
I20260925 01:29:18.141275 30488 raft_consensus.cc:1261] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Refusing update from remote peer 013664afced241cd81da058a46acb28a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:18.141459 30351 raft_consensus.cc:1261] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Refusing update from remote peer 013664afced241cd81da058a46acb28a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:18.141903 30613 consensus_queue.cc:1035] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:18.142211 30630 consensus_queue.cc:1035] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:18.194953 30047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:18.195186 30047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:18.198410 30047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: Sending DeleteTablet for 3 replicas of tablet 94f02f8148f248e1adab00209e34fae6
I20260925 01:29:18.199424 30465 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f02f8148f248e1adab00209e34fae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:18 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:58446
I20260925 01:29:18.199424 30326 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f02f8148f248e1adab00209e34fae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:18 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:46932
I20260925 01:29:18.199716 30637 tablet_replica.cc:330] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: stopping tablet replica
I20260925 01:29:18.199728 30636 tablet_replica.cc:330] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: stopping tablet replica
I20260925 01:29:18.199445 30185 tablet_service.cc:1514] Processing DeleteTablet for tablet 94f02f8148f248e1adab00209e34fae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:18 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:39892
I20260925 01:29:18.199944 30638 tablet_replica.cc:330] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: stopping tablet replica
I20260925 01:29:18.199944 30637 raft_consensus.cc:2229] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:18.199968 30636 raft_consensus.cc:2229] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:18.200142 30638 raft_consensus.cc:2229] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:18.200168 30637 raft_consensus.cc:2258] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:18.200237 30636 raft_consensus.cc:2258] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:18.200696 30638 raft_consensus.cc:2258] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:18.201447 30637 ts_tablet_manager.cc:1905] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:18.201992 30636 ts_tablet_manager.cc:1905] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:18.206202 30638 ts_tablet_manager.cc:1905] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:18.206599 30047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:18.207300 30047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:18.208621 30636 ts_tablet_manager.cc:1918] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:18.208770 30636 log.cc:1196] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/94f02f8148f248e1adab00209e34fae6
I20260925 01:29:18.209213 30636 ts_tablet_manager.cc:1939] T 94f02f8148f248e1adab00209e34fae6 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting consensus metadata
I20260925 01:29:18.210862 30032 catalog_manager.cc:4841] TS 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803): tablet 94f02f8148f248e1adab00209e34fae6 (table test-table [id=6a4e23b6038b431392ace720941924ba]) successfully deleted
I20260925 01:29:18.211050 30637 ts_tablet_manager.cc:1918] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:18.211217 30637 log.cc:1196] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/94f02f8148f248e1adab00209e34fae6
I20260925 01:29:18.211679 30637 ts_tablet_manager.cc:1939] T 94f02f8148f248e1adab00209e34fae6 P bb5781d682be4d96924dae671f6e68ce: Deleting consensus metadata
I20260925 01:29:18.213987 30034 catalog_manager.cc:4841] TS bb5781d682be4d96924dae671f6e68ce (127.27.160.2:40597): tablet 94f02f8148f248e1adab00209e34fae6 (table test-table [id=6a4e23b6038b431392ace720941924ba]) successfully deleted
I20260925 01:29:18.215646 30638 ts_tablet_manager.cc:1918] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:29:18.215873 30638 log.cc:1196] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/94f02f8148f248e1adab00209e34fae6
I20260925 01:29:18.215777 30465 tablet_service.cc:1467] Processing CreateTablet for tablet 6123415a8a68460597dab679c84567b5 (DEFAULT_TABLE table=test-table [id=40230533b5164dbbbc64bbbf0fe21f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:18.216241 30465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6123415a8a68460597dab679c84567b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:18.216187 30326 tablet_service.cc:1467] Processing CreateTablet for tablet 6123415a8a68460597dab679c84567b5 (DEFAULT_TABLE table=test-table [id=40230533b5164dbbbc64bbbf0fe21f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:18.216586 30638 ts_tablet_manager.cc:1939] T 94f02f8148f248e1adab00209e34fae6 P 013664afced241cd81da058a46acb28a: Deleting consensus metadata
I20260925 01:29:18.216607 30326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6123415a8a68460597dab679c84567b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:18.219621 30185 tablet_service.cc:1467] Processing CreateTablet for tablet 6123415a8a68460597dab679c84567b5 (DEFAULT_TABLE table=test-table [id=40230533b5164dbbbc64bbbf0fe21f9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:18.220511 30185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6123415a8a68460597dab679c84567b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:18.220568 30034 catalog_manager.cc:4841] TS 013664afced241cd81da058a46acb28a (127.27.160.1:46579): tablet 94f02f8148f248e1adab00209e34fae6 (table test-table [id=6a4e23b6038b431392ace720941924ba]) successfully deleted
I20260925 01:29:18.222443 30647 tablet_bootstrap.cc:492] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: Bootstrap starting.
I20260925 01:29:18.223353 30649 tablet_bootstrap.cc:492] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce: Bootstrap starting.
I20260925 01:29:18.224105 30647 tablet_bootstrap.cc:654] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:18.224915 30649 tablet_bootstrap.cc:654] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:18.231523 30648 tablet_bootstrap.cc:492] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a: Bootstrap starting.
I20260925 01:29:18.233349 30648 tablet_bootstrap.cc:654] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:18.235602 30649 tablet_bootstrap.cc:492] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce: No bootstrap required, opened a new log
I20260925 01:29:18.235893 30649 ts_tablet_manager.cc:1397] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260925 01:29:18.236970 30649 raft_consensus.cc:348] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.238814 30649 raft_consensus.cc:374] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:18.238979 30649 raft_consensus.cc:729] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5781d682be4d96924dae671f6e68ce, State: Initialized, Role: FOLLOWER
I20260925 01:29:18.239418 30649 consensus_queue.cc:260] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.240197 30649 ts_tablet_manager.cc:1428] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260925 01:29:18.241935 30647 tablet_bootstrap.cc:492] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: No bootstrap required, opened a new log
I20260925 01:29:18.242182 30647 ts_tablet_manager.cc:1397] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20260925 01:29:18.242997 30647 raft_consensus.cc:348] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.243840 30647 raft_consensus.cc:374] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:18.243984 30647 raft_consensus.cc:729] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e04f723ffc84d19a9b773129d6d93a6, State: Initialized, Role: FOLLOWER
I20260925 01:29:18.244491 30647 consensus_queue.cc:260] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.245294 30647 ts_tablet_manager.cc:1428] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 01:29:18.249668 30648 tablet_bootstrap.cc:492] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a: No bootstrap required, opened a new log
I20260925 01:29:18.249855 30648 ts_tablet_manager.cc:1397] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.012s
I20260925 01:29:18.250821 30648 raft_consensus.cc:348] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.251091 30648 raft_consensus.cc:374] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:18.251179 30648 raft_consensus.cc:729] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013664afced241cd81da058a46acb28a, State: Initialized, Role: FOLLOWER
I20260925 01:29:18.251441 30648 consensus_queue.cc:260] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.252015 30648 ts_tablet_manager.cc:1428] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 01:29:18.463362 30631 raft_consensus.cc:482] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:18.463581 30631 raft_consensus.cc:504] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.464254 30631 leader_election.cc:288] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 013664afced241cd81da058a46acb28a (127.27.160.1:46579), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:18.464902 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6123415a8a68460597dab679c84567b5" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6" is_pre_election: true
I20260925 01:29:18.464964 30205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6123415a8a68460597dab679c84567b5" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013664afced241cd81da058a46acb28a" is_pre_election: true
I20260925 01:29:18.465219 30490 raft_consensus.cc:2454] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 0.
I20260925 01:29:18.465303 30205 raft_consensus.cc:2454] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 0.
I20260925 01:29:18.465737 30276 leader_election.cc:302] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e04f723ffc84d19a9b773129d6d93a6, bb5781d682be4d96924dae671f6e68ce; no voters:
I20260925 01:29:18.465997 30631 raft_consensus.cc:2781] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:18.466120 30631 raft_consensus.cc:482] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:18.466245 30631 raft_consensus.cc:3037] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:18.468241 30631 raft_consensus.cc:504] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.468798 30631 leader_election.cc:288] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 election: Requested vote from peers 013664afced241cd81da058a46acb28a (127.27.160.1:46579), 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803)
I20260925 01:29:18.469408 30490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6123415a8a68460597dab679c84567b5" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e04f723ffc84d19a9b773129d6d93a6"
I20260925 01:29:18.469521 30205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6123415a8a68460597dab679c84567b5" candidate_uuid: "bb5781d682be4d96924dae671f6e68ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013664afced241cd81da058a46acb28a"
I20260925 01:29:18.469657 30490 raft_consensus.cc:3037] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:18.469731 30205 raft_consensus.cc:3037] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:18.472752 30490 raft_consensus.cc:2454] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 1.
I20260925 01:29:18.472924 30205 raft_consensus.cc:2454] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb5781d682be4d96924dae671f6e68ce in term 1.
I20260925 01:29:18.473364 30276 leader_election.cc:302] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e04f723ffc84d19a9b773129d6d93a6, bb5781d682be4d96924dae671f6e68ce; no voters:
I20260925 01:29:18.473683 30631 raft_consensus.cc:2781] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:18.474121 30631 raft_consensus.cc:686] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 1 LEADER]: Becoming Leader. State: Replica: bb5781d682be4d96924dae671f6e68ce, State: Running, Role: LEADER
I20260925 01:29:18.474416 30631 consensus_queue.cc:237] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } }
I20260925 01:29:18.477125 30046 catalog_manager.cc:5495] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce reported cstate change: term changed from 0 to 1, leader changed from <none> to bb5781d682be4d96924dae671f6e68ce (127.27.160.2). New cstate: current_term: 1 leader_uuid: "bb5781d682be4d96924dae671f6e68ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb5781d682be4d96924dae671f6e68ce" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 40597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260925 01:29:18.537292 30671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:18.531529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.27.160.2:40597)
0925 01:29:18.532111 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:18.532116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:18.532130 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:29:18.532181 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:18.532184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:18.532244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:18.532351 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:18.532770 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:18.533654 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:18.535002 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:18.535406 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:18.535539 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:18.535999 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:18.536121 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:18.536518 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:18.536699 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 01:29:18.536810 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":304,"threads_started":1}
I20260925 01:29:18.547818 30205 raft_consensus.cc:1261] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Refusing update from remote peer bb5781d682be4d96924dae671f6e68ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:18.547832 30490 raft_consensus.cc:1261] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Refusing update from remote peer bb5781d682be4d96924dae671f6e68ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:18.548416 30668 consensus_queue.cc:1035] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e04f723ffc84d19a9b773129d6d93a6" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:18.548765 30631 consensus_queue.cc:1035] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "013664afced241cd81da058a46acb28a" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:29:18.575387 30662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 01:29:20.203332 30046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:20.203644 30046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:20.208324 30046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6288b8aa0da74e91a99507448d218b5c: Sending DeleteTablet for 3 replicas of tablet 6123415a8a68460597dab679c84567b5
I20260925 01:29:20.209239 30185 tablet_service.cc:1514] Processing DeleteTablet for tablet 6123415a8a68460597dab679c84567b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:20 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:39892
I20260925 01:29:20.209287 30326 tablet_service.cc:1514] Processing DeleteTablet for tablet 6123415a8a68460597dab679c84567b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:20 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:46932
I20260925 01:29:20.209288 30465 tablet_service.cc:1514] Processing DeleteTablet for tablet 6123415a8a68460597dab679c84567b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:20 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:58446
I20260925 01:29:20.209956 30704 tablet_replica.cc:330] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce: stopping tablet replica
I20260925 01:29:20.210316 30704 raft_consensus.cc:2229] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:20.210482 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30120
I20260925 01:29:20.210927 30704 raft_consensus.cc:2258] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:20.212321 30704 ts_tablet_manager.cc:1905] T 6123415a8a68460597dab679c84567b5 P bb5781d682be4d96924dae671f6e68ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:20.218289 30705 tablet_replica.cc:330] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a: stopping tablet replica
I20260925 01:29:20.218592 30705 raft_consensus.cc:2229] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:20.219000 30705 raft_consensus.cc:2258] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:20.219056 30706 tablet_replica.cc:330] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: stopping tablet replica
I20260925 01:29:20.219754 30706 raft_consensus.cc:2229] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:20.220039 30706 raft_consensus.cc:2258] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:20.220064 30705 ts_tablet_manager.cc:1905] T 6123415a8a68460597dab679c84567b5 P 013664afced241cd81da058a46acb28a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:20.221105 30706 ts_tablet_manager.cc:1905] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 01:29:20 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:20.244192 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.213238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.27.160.1:46579)
0925 01:29:20.219216 (+ 5978us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:20.219223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:20.219247 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:29:20.219318 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:20.219323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:20.219431 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:20.219567 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:20.221235 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:20.222585 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:20.224114 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:20.224525 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:20.224635 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:20.237051 (+ 12416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:20.237070 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:20.237087 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:20.237769 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:20.240892 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:20.241861 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:20.241869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:20.241872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:20.242053 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:20.242525 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:20.242533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:20.242536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:20.242923 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:20.243250 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:20.243583 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 01:29:20.243812 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5856,"thread_start_us":5674,"threads_started":1}
W20260925 01:29:20.244868 30251 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39922
W20260925 01:29:20.245579 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30120: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:20.245779 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 30120
I20260925 01:29:20.284948 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30260
W20260925 01:29:20.285048 30034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.160.1:46579 (error 108)
I20260925 01:29:20.287791 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.286591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.0.0.1:49440)
0925 01:29:20.287191 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.287455 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":308,"threads_started":1}
W20260925 01:29:20.288057 30034 catalog_manager.cc:4584] TS 013664afced241cd81da058a46acb28a (127.27.160.1:46579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6123415a8a68460597dab679c84567b5: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46579: connect: Connection refused (error 111)
Sep 25 01:29:20 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:29:20.304704 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.285778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.27.160.2:40597)
0925 01:29:20.286510 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:20.286517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:20.286537 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:29:20.286604 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:20.286609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:20.286691 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:20.286845 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:20.288838 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:20.289746 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:20.291379 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:20.292225 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:20.292354 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:20.297684 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:20.297702 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:20.297721 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:20.297779 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:20.301550 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:20.302356 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:20.302364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:20.302367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:20.302524 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:20.302950 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:20.302954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:20.302956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:20.303359 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:20.303651 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:20.303867 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 01:29:20.304210 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":411,"threads_started":1}
W20260925 01:29:20.305429 30391 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54496
W20260925 01:29:20.306075 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30260: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:20.306242 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 30260
I20260925 01:29:20.342121 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.341608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.0.0.1:49442)
0925 01:29:20.341837 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.341938 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 01:29:20.343279 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.342851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.0.0.1:49454)
0925 01:29:20.343056 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.343152 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 01:29:20.362327 30706 ts_tablet_manager.cc:1918] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:29:20.362550 30706 log.cc:1196] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/6123415a8a68460597dab679c84567b5
I20260925 01:29:20.363050 30706 ts_tablet_manager.cc:1939] T 6123415a8a68460597dab679c84567b5 P 7e04f723ffc84d19a9b773129d6d93a6: Deleting consensus metadata
I20260925 01:29:20.364498 30032 catalog_manager.cc:4841] TS 7e04f723ffc84d19a9b773129d6d93a6 (127.27.160.3:40803): tablet 6123415a8a68460597dab679c84567b5 (table test-table [id=40230533b5164dbbbc64bbbf0fe21f9b]) successfully deleted
I20260925 01:29:20.383216 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30400
I20260925 01:29:20.384265 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.383735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:40597 (local address 127.0.0.1:54512)
0925 01:29:20.383920 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.384036 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.2:40597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
Sep 25 01:29:20 dist-test-slave-h6hf krb5kdc[30003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:29:20.400365 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.384059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.27.160.3:40803)
0925 01:29:20.384719 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:20.384723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:20.384739 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:29:20.384792 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:20.384796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:20.384860 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:20.385019 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:20.386455 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:20.387356 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:20.388691 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:20.389425 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:20.389547 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:20.394352 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:20.394371 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:20.394390 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:20.394434 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:20.397540 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:20.398388 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:20.398397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:20.398399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:20.398564 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:20.398924 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:20.398927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:20.398929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:20.399282 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:20.399541 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:20.399783 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 01:29:20.399979 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":390,"threads_started":1}
W20260925 01:29:20.401311 30530 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47740
W20260925 01:29:20.402119 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30400: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:20.402336 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 30400
I20260925 01:29:20.404228 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.403634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.0.0.1:49462)
0925 01:29:20.403883 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.404001 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 01:29:20.405409 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.404974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.0.0.1:49466)
0925 01:29:20.405221 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.405292 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 01:29:20.417040 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.416554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:40597 (local address 127.0.0.1:54528)
0925 01:29:20.416787 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.416876 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.2:40597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 01:29:20.418272 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.417870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:40597 (local address 127.0.0.1:54542)
0925 01:29:20.418046 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.418149 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.2:40597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 01:29:20.421300 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30016
I20260925 01:29:20.422580 30088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:20.547124 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.546370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:40597 (local address 127.0.0.1:54554)
0925 01:29:20.546737 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.546871 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.2:40597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260925 01:29:20.547590 30711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:20.547144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46579 (local address 127.0.0.1:49470)
0925 01:29:20.547376 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:20.547455 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"threads_started":1}
I20260925 01:29:20.549424 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 30016
2026-09-25T01:29:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8878 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:22 dist-test-slave-h6hf krb5kdc[30717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:22 dist-test-slave-h6hf krb5kdc[30717](info): set up 2 sockets
Sep 25 01:29:22 dist-test-slave-h6hf krb5kdc[30717](info): commencing operation
krb5kdc: starting...
W20260925 01:29:23.862442 28288 mini_kdc.cc:121] Time spent starting KDC: real 3.266s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:23 dist-test-slave-h6hf krb5kdc[30717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:24.069322 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:39057
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:39057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:24.393734 30730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:24.394042 30730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:24.394196 30730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:24.401698 30730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:24.401824 30730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:24.401934 30730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:24.402045 30730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:24.402130 30730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:24.412441 30730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:39057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:39057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:24.413918 30730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:24.415262 30730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:24.428793 30735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:24.428920 30736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:24.429626 30730 server_base.cc:1034] running on GCE node
W20260925 01:29:24.429631 30738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:24.430588 30730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:24.432133 30730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:24.433375 30730 hybrid_clock.cc:648] HybridClock initialized: now 1790299764433316 us; error 51 us; skew 500 ppm
I20260925 01:29:24.433754 30730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:24 dist-test-slave-h6hf krb5kdc[30717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299764, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:24.437289 30730 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:24.438843 30730 webserver.cc:460] Webserver started at http://127.27.160.62:39639/ using document root <none> and password file <none>
I20260925 01:29:24.439561 30730 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:24.439710 30730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:24.440039 30730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:24.443104 30730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "e9dfe2950d0f4b168f10e5f7a613a34f"
format_stamp: "Formatted at 2026-09-25 01:29:24 on dist-test-slave-h6hf"
server_key: "c4b1a9073cfc1fd48e9623ba992dd1ec"
server_key_iv: "e4321d6cb9a6d56bf636ee90dcfe86e7"
server_key_version: "encryptionkey@0"
I20260925 01:29:24.443856 30730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "e9dfe2950d0f4b168f10e5f7a613a34f"
format_stamp: "Formatted at 2026-09-25 01:29:24 on dist-test-slave-h6hf"
server_key: "c4b1a9073cfc1fd48e9623ba992dd1ec"
server_key_iv: "e4321d6cb9a6d56bf636ee90dcfe86e7"
server_key_version: "encryptionkey@0"
I20260925 01:29:24.449010 30730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:29:24.452527 30745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:24.454031 30730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:24.454231 30730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "e9dfe2950d0f4b168f10e5f7a613a34f"
format_stamp: "Formatted at 2026-09-25 01:29:24 on dist-test-slave-h6hf"
server_key: "c4b1a9073cfc1fd48e9623ba992dd1ec"
server_key_iv: "e4321d6cb9a6d56bf636ee90dcfe86e7"
server_key_version: "encryptionkey@0"
I20260925 01:29:24.454439 30730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:24.480856 30730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:24.485718 30730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:24.486042 30730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:24.505093 30730 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:39057
I20260925 01:29:24.505180 30796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:39057 every 8 connection(s)
I20260925 01:29:24.506561 30730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:24.508641 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 30730
I20260925 01:29:24.508884 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:24.509706 28288 external_mini_cluster.cc:1428] Setting key ee9b832d16d635fea4bc0990b307fbc6
I20260925 01:29:24.511225 30797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:29:24 dist-test-slave-h6hf krb5kdc[30717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:24.523999 30797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f: Bootstrap starting.
I20260925 01:29:24.527572 30797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:24.528659 30797 log.cc:824] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:24.531780 30797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f: No bootstrap required, opened a new log
I20260925 01:29:24.532305 30800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:24.511891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.27.160.62:39057)
0925 01:29:24.512679 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:24.512690 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:24.512733 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:29:24.514594 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:24.514619 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:24.515041 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:24.515493 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:24.517030 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:24.518456 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:24.519708 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:24.520390 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:24.520647 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:24.524856 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:24.524864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:24.524879 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:24.524914 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:24.528126 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:24.528857 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:24.528864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:24.528870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:24.529043 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:24.529447 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:24.529453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:24.529456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:24.530100 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:24.530390 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:24.530853 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 01:29:24.531283 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":350,"threads_started":1}
I20260925 01:29:24.539760 30797 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dfe2950d0f4b168f10e5f7a613a34f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39057 } }
I20260925 01:29:24.540262 30797 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:24.540375 30797 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9dfe2950d0f4b168f10e5f7a613a34f, State: Initialized, Role: FOLLOWER
I20260925 01:29:24.541049 30797 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dfe2950d0f4b168f10e5f7a613a34f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39057 } }
I20260925 01:29:24.541378 30797 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:24.541540 30797 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:24.541730 30797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:24.543889 30797 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dfe2950d0f4b168f10e5f7a613a34f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39057 } }
I20260925 01:29:24.544638 30797 leader_election.cc:302] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9dfe2950d0f4b168f10e5f7a613a34f; no voters:
I20260925 01:29:24.545343 30797 leader_election.cc:288] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:24.545543 30802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:24.546811 30802 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [term 1 LEADER]: Becoming Leader. State: Replica: e9dfe2950d0f4b168f10e5f7a613a34f, State: Running, Role: LEADER
I20260925 01:29:24.547583 30802 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dfe2950d0f4b168f10e5f7a613a34f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39057 } }
I20260925 01:29:24.547734 30797 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:24.551474 30804 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9dfe2950d0f4b168f10e5f7a613a34f. Latest consensus state: current_term: 1 leader_uuid: "e9dfe2950d0f4b168f10e5f7a613a34f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dfe2950d0f4b168f10e5f7a613a34f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39057 } } }
I20260925 01:29:24.551365 30803 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9dfe2950d0f4b168f10e5f7a613a34f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dfe2950d0f4b168f10e5f7a613a34f" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39057 } } }
I20260925 01:29:24.552029 30804 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:24.552055 30803 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:24.555612 30811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:24.561277 30811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:24.570834 30811 catalog_manager.cc:1347] Generated new cluster ID: 13c5f2b05d4b47948d6112df0156ef15
I20260925 01:29:24.571027 30811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:24.589002 30811 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:24.590945 30811 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:24.602550 30811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9dfe2950d0f4b168f10e5f7a613a34f: Generated new TSK 0
I20260925 01:29:24.603446 30811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:29:24.662891 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39057
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:25.008468 30823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:25.008987 30823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:25.009171 30823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:25.017292 30823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:25.017411 30823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:25.017529 30823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:25.027783 30823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:25.029085 30823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:25.030527 30823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:25.044718 30828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:25.045290 30829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:25.045368 30831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:25.046464 30823 server_base.cc:1034] running on GCE node
I20260925 01:29:25.047135 30823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:25.048193 30823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:25.049431 30823 hybrid_clock.cc:648] HybridClock initialized: now 1790299765049358 us; error 65 us; skew 500 ppm
I20260925 01:29:25.049795 30823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:25 dist-test-slave-h6hf krb5kdc[30717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299765, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:25.053093 30823 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:29:25.054919 30823 webserver.cc:460] Webserver started at http://127.27.160.1:44553/ using document root <none> and password file <none>
I20260925 01:29:25.055742 30823 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:25.055891 30823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:25.056229 30823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:25.059587 30823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "4675e56db8eb48bc8b14498bb32effd9"
format_stamp: "Formatted at 2026-09-25 01:29:25 on dist-test-slave-h6hf"
server_key: "ee7391644ad16dbea055a152a4f7f565"
server_key_iv: "aa7f87d330b12d12786e0d6f0d6065ae"
server_key_version: "encryptionkey@0"
I20260925 01:29:25.060421 30823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "4675e56db8eb48bc8b14498bb32effd9"
format_stamp: "Formatted at 2026-09-25 01:29:25 on dist-test-slave-h6hf"
server_key: "ee7391644ad16dbea055a152a4f7f565"
server_key_iv: "aa7f87d330b12d12786e0d6f0d6065ae"
server_key_version: "encryptionkey@0"
I20260925 01:29:25.066016 30823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:29:25.069819 30838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:25.071307 30823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:25.071509 30823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "4675e56db8eb48bc8b14498bb32effd9"
format_stamp: "Formatted at 2026-09-25 01:29:25 on dist-test-slave-h6hf"
server_key: "ee7391644ad16dbea055a152a4f7f565"
server_key_iv: "aa7f87d330b12d12786e0d6f0d6065ae"
server_key_version: "encryptionkey@0"
I20260925 01:29:25.071738 30823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:25.082978 30823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:25.087409 30823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:25.087721 30823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:25.088721 30823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:25.090229 30823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:25.090334 30823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:25.090488 30823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:25.090564 30823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:25.124671 30823 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:42605
I20260925 01:29:25.124718 30950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:42605 every 8 connection(s)
I20260925 01:29:25.126060 30823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:25.135737 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 30823
I20260925 01:29:25.135988 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:25.136684 28288 external_mini_cluster.cc:1428] Setting key c459bb4e60fb47948a7f8b788edddf4f
Sep 25 01:29:25 dist-test-slave-h6hf krb5kdc[30717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299765, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:25.146579 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:25.128737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:60381 (local address 127.27.160.62:39057)
0925 01:29:25.129218 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:25.129223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:25.130171 (+ 948us) server_negotiation.cc:407] Connection header received
0925 01:29:25.131940 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:25.131944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:25.132012 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:25.132111 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:25.133677 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:25.134386 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:25.136341 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:25.136780 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:25.136936 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:25.140740 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:25.140746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:25.140749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:25.140780 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:25.143444 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:25.144205 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:25.144215 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:25.144217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:25.144331 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:25.144775 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:25.144780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:25.144784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:25.145051 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:25.145256 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:25.145982 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 01:29:25.146219 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":210,"threads_started":1}
I20260925 01:29:25.147718 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:25.129316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39057 (local address 127.27.160.1:60381)
0925 01:29:25.130000 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:25.130042 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:25.131675 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:25.132307 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:25.132314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:25.132856 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:25.133513 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:25.133530 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:25.134529 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:25.134532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:25.136188 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:25.136211 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:25.137024 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:25.137033 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:25.137372 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:25.138197 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:25.138219 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:25.140510 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:25.143601 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:25.143611 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:25.143629 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:25.144074 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:25.144444 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:25.144449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:25.144453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:25.144654 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:25.145236 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:25.145245 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:25.145603 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 01:29:25.145874 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 01:29:25.146418 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":258,"threads_started":1}
I20260925 01:29:25.149392 30951 heartbeater.cc:347] Connected to a master server at 127.27.160.62:39057
I20260925 01:29:25.149750 30951 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:25.150655 30951 heartbeater.cc:511] Master 127.27.160.62:39057 requested a full tablet report, sending...
I20260925 01:29:25.153275 30762 ts_manager.cc:194] Registered new tserver with Master: 4675e56db8eb48bc8b14498bb32effd9 (127.27.160.1:42605)
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:29:25.155259 30762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:60381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:25.180891 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39057
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:25.513907 30957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:25.514195 30957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:25.514367 30957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:25.522068 30957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:25.522218 30957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:25.522410 30957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:25.532624 30957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:25.534121 30957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:25.535847 30957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:25.549966 30962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:25.549983 30963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:25.550077 30965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:25.550762 30957 server_base.cc:1034] running on GCE node
I20260925 01:29:25.551326 30957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:25.552277 30957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:25.553506 30957 hybrid_clock.cc:648] HybridClock initialized: now 1790299765553451 us; error 49 us; skew 500 ppm
I20260925 01:29:25.553846 30957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:25 dist-test-slave-h6hf krb5kdc[30717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299765, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:25.563185 30957 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:25.564865 30957 webserver.cc:460] Webserver started at http://127.27.160.2:44437/ using document root <none> and password file <none>
I20260925 01:29:25.565820 30957 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:25.565980 30957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:25.566316 30957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:25.569466 30957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "5a92adfb0695447b96ab162507424c38"
format_stamp: "Formatted at 2026-09-25 01:29:25 on dist-test-slave-h6hf"
server_key: "4c9e69be9fcc185b4d1d3b449006c231"
server_key_iv: "9dfb264fcfe4085badf4de4ad6d19d79"
server_key_version: "encryptionkey@0"
I20260925 01:29:25.570291 30957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "5a92adfb0695447b96ab162507424c38"
format_stamp: "Formatted at 2026-09-25 01:29:25 on dist-test-slave-h6hf"
server_key: "4c9e69be9fcc185b4d1d3b449006c231"
server_key_iv: "9dfb264fcfe4085badf4de4ad6d19d79"
server_key_version: "encryptionkey@0"
I20260925 01:29:25.575874 30957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:29:25.579722 30972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:25.581239 30957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:29:25.581422 30957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "5a92adfb0695447b96ab162507424c38"
format_stamp: "Formatted at 2026-09-25 01:29:25 on dist-test-slave-h6hf"
server_key: "4c9e69be9fcc185b4d1d3b449006c231"
server_key_iv: "9dfb264fcfe4085badf4de4ad6d19d79"
server_key_version: "encryptionkey@0"
I20260925 01:29:25.581653 30957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:25.612841 30957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:25.617313 30957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:25.617616 30957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:25.618587 30957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:25.619920 30957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:25.620030 30957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:25.620177 30957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:25.620258 30957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:25.655787 30957 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:32829
I20260925 01:29:25.655804 31084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:32829 every 8 connection(s)
I20260925 01:29:25.657377 30957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:25.664482 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 30957
I20260925 01:29:25.664717 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:25.665248 28288 external_mini_cluster.cc:1428] Setting key 66b44394b5e632716737116eba2ce81b
Sep 25 01:29:25 dist-test-slave-h6hf krb5kdc[30717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299765, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:25.679661 31087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:25.660295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:40049 (local address 127.27.160.62:39057)
0925 01:29:25.660733 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:25.660738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:25.661749 (+ 1011us) server_negotiation.cc:407] Connection header received
0925 01:29:25.663564 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:25.663569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:25.663663 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:25.663781 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:25.665466 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:25.666481 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:25.668310 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:25.669278 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:25.669441 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:25.673327 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:25.673334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:25.673337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:25.673374 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:25.676103 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:25.677187 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:25.677193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:25.677203 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:25.677333 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:25.677882 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:25.677885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:25.677888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:25.678157 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:25.678255 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:25.679227 (+ 972us) server_negotiation.cc:299] Negotiation successful
0925 01:29:25.679400 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":256,"threads_started":1}
I20260925 01:29:25.681485 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:25.660824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39057 (local address 127.27.160.2:40049)
0925 01:29:25.661585 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:25.661627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:25.663290 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:25.664016 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:25.664024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:25.664535 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:25.665267 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:25.665287 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:25.666640 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:25.666644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:25.668139 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:25.668153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:25.669505 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:25.669509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:25.669739 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:25.670497 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:25.670522 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:25.672972 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:25.676286 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:25.676298 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:25.676321 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:25.676995 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:25.677459 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:25.677467 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:25.677473 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:25.677768 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:25.678296 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:25.678308 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:25.678823 (+ 515us) client_negotiation.cc:769] Sending connection context
0925 01:29:25.679159 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 01:29:25.679804 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":353,"threads_started":1}
I20260925 01:29:25.683149 31085 heartbeater.cc:347] Connected to a master server at 127.27.160.62:39057
I20260925 01:29:25.683524 31085 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:25.684497 31085 heartbeater.cc:511] Master 127.27.160.62:39057 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:29:25.686466 30762 ts_manager.cc:194] Registered new tserver with Master: 5a92adfb0695447b96ab162507424c38 (127.27.160.2:32829)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
I20260925 01:29:25.687742 30762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:40049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:25.712576 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39057
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:26.045315 31091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:26.045626 31091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:26.045722 31091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:26.053318 31091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:26.053433 31091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:26.053551 31091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:26.063480 31091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:26.064780 31091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:26.066063 31091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:26.079778 31097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:26.079867 31099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:26.079815 31096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:26.080724 31091 server_base.cc:1034] running on GCE node
I20260925 01:29:26.081481 31091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:26.082525 31091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:26.083756 31091 hybrid_clock.cc:648] HybridClock initialized: now 1790299766083712 us; error 80 us; skew 500 ppm
I20260925 01:29:26.084090 31091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:26 dist-test-slave-h6hf krb5kdc[30717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299766, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:26.087670 31091 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:26.089326 31091 webserver.cc:460] Webserver started at http://127.27.160.3:37367/ using document root <none> and password file <none>
I20260925 01:29:26.090052 31091 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:26.090199 31091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:26.090529 31091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:26.093638 31091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "3679ce75c0144a55b354eb95f507991d"
format_stamp: "Formatted at 2026-09-25 01:29:26 on dist-test-slave-h6hf"
server_key: "f53e3e9a837caa0967697fccc0e12b33"
server_key_iv: "802c4336eadcb60fd47594cb89066082"
server_key_version: "encryptionkey@0"
I20260925 01:29:26.094506 31091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "3679ce75c0144a55b354eb95f507991d"
format_stamp: "Formatted at 2026-09-25 01:29:26 on dist-test-slave-h6hf"
server_key: "f53e3e9a837caa0967697fccc0e12b33"
server_key_iv: "802c4336eadcb60fd47594cb89066082"
server_key_version: "encryptionkey@0"
I20260925 01:29:26.100232 31091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:29:26.104321 31106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:26.105937 31091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:26.106144 31091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "3679ce75c0144a55b354eb95f507991d"
format_stamp: "Formatted at 2026-09-25 01:29:26 on dist-test-slave-h6hf"
server_key: "f53e3e9a837caa0967697fccc0e12b33"
server_key_iv: "802c4336eadcb60fd47594cb89066082"
server_key_version: "encryptionkey@0"
I20260925 01:29:26.106371 31091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:26.146905 31091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:26.151603 31091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:26.151914 31091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:26.152948 31091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:26.154384 31091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:26.154501 31091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:26.154646 31091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:26.154726 31091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:26.158730 30951 heartbeater.cc:503] Master 127.27.160.62:39057 was elected leader, sending a full tablet report...
I20260925 01:29:26.193765 31091 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:38145
I20260925 01:29:26.193811 31218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:38145 every 8 connection(s)
I20260925 01:29:26.195147 31091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:26.202324 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 31091
I20260925 01:29:26.202554 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:26.203028 28288 external_mini_cluster.cc:1428] Setting key df1414b0a95680234d4355e6eacb0119
Sep 25 01:29:26 dist-test-slave-h6hf krb5kdc[30717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299766, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:26.217940 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:26.198161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:40749 (local address 127.27.160.62:39057)
0925 01:29:26.198644 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:26.198649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:26.199443 (+ 794us) server_negotiation.cc:407] Connection header received
0925 01:29:26.202484 (+ 3041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:26.202490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:26.202590 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:26.202713 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:26.204951 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:26.205902 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:26.207479 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:26.207874 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:26.208011 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:26.211922 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:26.211931 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:26.211937 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:26.211980 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:26.214720 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:26.215535 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:26.215541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:26.215545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:26.215673 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:26.216119 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:26.216124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:26.216128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:26.216402 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:26.216563 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:26.217282 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 01:29:26.217494 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":254,"threads_started":1}
I20260925 01:29:26.219022 31222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:26.198583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39057 (local address 127.27.160.3:40749)
0925 01:29:26.199265 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:26.199306 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:26.202128 (+ 2822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:26.203095 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:26.203103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:26.203875 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:26.204760 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:26.204782 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:26.206056 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:26.206061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:26.207322 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:26.207334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:26.208112 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:26.208120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:26.208353 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:26.209089 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:26.209112 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:26.211660 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:26.214873 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:26.214881 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:26.214900 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:26.215345 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:26.215787 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:26.215792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:26.215795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:26.216003 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:26.216537 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:26.216545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:26.216894 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 01:29:26.217214 (+ 320us) client_negotiation.cc:240] Negotiation successful
0925 01:29:26.217760 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":264,"threads_started":1}
I20260925 01:29:26.220566 31219 heartbeater.cc:347] Connected to a master server at 127.27.160.62:39057
I20260925 01:29:26.220840 31219 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:26.221635 31219 heartbeater.cc:511] Master 127.27.160.62:39057 requested a full tablet report, sending...
I20260925 01:29:26.222987 30762 ts_manager.cc:194] Registered new tserver with Master: 3679ce75c0144a55b354eb95f507991d (127.27.160.3:38145)
I20260925 01:29:26.223841 30762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:40749
I20260925 01:29:26.232385 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 01:29:26.263851 31221 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:29:26.261368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.27.160.62:39057)
0925 01:29:26.261617 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:26.261626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:26.262057 (+ 431us) server_negotiation.cc:407] Connection header received
0925 01:29:26.262205 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:26.262209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:26.262293 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:26.262379 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:26.263587 (+ 1208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55860: BlockingRecv error: recv got EOF from 127.0.0.1:55860 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:29:26.267304 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30823
W20260925 01:29:26.270591 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30823: Not authorized: Client connection negotiation failed: client connection to 127.27.160.1:42605: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:29:26.270789 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 30823
I20260925 01:29:26.287137 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30957
W20260925 01:29:26.289908 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30957: Not authorized: Client connection negotiation failed: client connection to 127.27.160.2:32829: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:29:26.290084 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 30957
I20260925 01:29:26.305297 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31091
W20260925 01:29:26.308153 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31091: Not authorized: Client connection negotiation failed: client connection to 127.27.160.3:38145: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:29:26.308349 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 31091
I20260925 01:29:26.326028 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 30730
I20260925 01:29:26.327395 30792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:26.430943 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 30730
2026-09-25T01:29:26Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5876 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:26 dist-test-slave-h6hf krb5kdc[31237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:26 dist-test-slave-h6hf krb5kdc[31237](info): set up 2 sockets
Sep 25 01:29:26 dist-test-slave-h6hf krb5kdc[31237](info): commencing operation
krb5kdc: starting...
W20260925 01:29:27.540251 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:27 dist-test-slave-h6hf krb5kdc[31237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:27.745656 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:46381
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:44267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:46381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:28.092724 31250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:28.093127 31250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:28.093321 31250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:28.101405 31250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:28.101560 31250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:28.101637 31250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:28.101706 31250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:28.101758 31250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:28.112974 31250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:46381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:46381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:28.114562 31250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:28.115940 31250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:28.129165 31256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:28.129263 31255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:28.129261 31258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:28.130205 31250 server_base.cc:1034] running on GCE node
I20260925 01:29:28.130990 31250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:28.132486 31250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:28.133692 31250 hybrid_clock.cc:648] HybridClock initialized: now 1790299768133655 us; error 30 us; skew 500 ppm
I20260925 01:29:28.134028 31250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:28 dist-test-slave-h6hf krb5kdc[31237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299768, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:28.137300 31250 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:28.138772 31250 webserver.cc:460] Webserver started at http://127.27.160.62:44301/ using document root <none> and password file <none>
I20260925 01:29:28.139488 31250 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:28.139627 31250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:28.139963 31250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:28.143146 31250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "fc4406b96a79419783a2f98a4cba0aea"
format_stamp: "Formatted at 2026-09-25 01:29:28 on dist-test-slave-h6hf"
server_key: "6e32c4373570de52345a9e8095707e6e"
server_key_iv: "cf0dd9c9174f4ef07204683c31da903c"
server_key_version: "encryptionkey@0"
I20260925 01:29:28.143992 31250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "fc4406b96a79419783a2f98a4cba0aea"
format_stamp: "Formatted at 2026-09-25 01:29:28 on dist-test-slave-h6hf"
server_key: "6e32c4373570de52345a9e8095707e6e"
server_key_iv: "cf0dd9c9174f4ef07204683c31da903c"
server_key_version: "encryptionkey@0"
I20260925 01:29:28.149493 31250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:29:28.152894 31265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:28.154383 31250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:28.154592 31250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "fc4406b96a79419783a2f98a4cba0aea"
format_stamp: "Formatted at 2026-09-25 01:29:28 on dist-test-slave-h6hf"
server_key: "6e32c4373570de52345a9e8095707e6e"
server_key_iv: "cf0dd9c9174f4ef07204683c31da903c"
server_key_version: "encryptionkey@0"
I20260925 01:29:28.154799 31250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:28.171737 31250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:28.176237 31250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:28.176548 31250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:28.196381 31250 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:46381
I20260925 01:29:28.196396 31316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:46381 every 8 connection(s)
I20260925 01:29:28.197883 31250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:28.201613 31317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:28.205710 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 31250
I20260925 01:29:28.205921 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:28.206557 28288 external_mini_cluster.cc:1428] Setting key 4418ee1d1f5af4781e70b4aabf5a5444
I20260925 01:29:28.214083 31317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea: Bootstrap starting.
I20260925 01:29:28.217867 31317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:28.219255 31317 log.cc:824] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:28.222486 31317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea: No bootstrap required, opened a new log
Sep 25 01:29:28 dist-test-slave-h6hf krb5kdc[31237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:28.232824 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:28.208417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.27.160.62:46381)
0925 01:29:28.209318 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:28.209339 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:28.209393 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:29:28.212262 (+ 2869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:28.212292 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:28.212816 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:28.213325 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:28.214750 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:28.216792 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:28.218160 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:28.219261 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:28.219463 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:28.223736 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:28.223747 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:28.223767 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:28.223816 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:28.227949 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:28.228757 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:28.228767 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:28.228776 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:28.229001 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:28.229406 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:28.229411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:28.229415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:28.230220 (+ 805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:28.230536 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:28.230877 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 01:29:28.231388 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":363,"threads_started":1}
I20260925 01:29:28.233541 31317 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4406b96a79419783a2f98a4cba0aea" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46381 } }
I20260925 01:29:28.233934 31317 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:28.234046 31317 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4406b96a79419783a2f98a4cba0aea, State: Initialized, Role: FOLLOWER
I20260925 01:29:28.234627 31317 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4406b96a79419783a2f98a4cba0aea" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46381 } }
I20260925 01:29:28.234879 31317 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:28.235030 31317 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:28.235224 31317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:28.236996 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:29:28.237881 31317 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4406b96a79419783a2f98a4cba0aea" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46381 } }
I20260925 01:29:28.238369 31317 leader_election.cc:302] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc4406b96a79419783a2f98a4cba0aea; no voters:
I20260925 01:29:28.238968 31317 leader_election.cc:288] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:28.239115 31322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:28.240924 31317 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:28.246171 31322 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [term 1 LEADER]: Becoming Leader. State: Replica: fc4406b96a79419783a2f98a4cba0aea, State: Running, Role: LEADER
I20260925 01:29:28.246776 31322 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4406b96a79419783a2f98a4cba0aea" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46381 } }
I20260925 01:29:28.249687 31323 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc4406b96a79419783a2f98a4cba0aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4406b96a79419783a2f98a4cba0aea" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46381 } } }
I20260925 01:29:28.250099 31323 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:28.250267 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc4406b96a79419783a2f98a4cba0aea. Latest consensus state: current_term: 1 leader_uuid: "fc4406b96a79419783a2f98a4cba0aea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4406b96a79419783a2f98a4cba0aea" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46381 } } }
I20260925 01:29:28.250541 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea [sys.catalog]: This master's current role is: LEADER
W20260925 01:29:28.263972 31337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:29:28.264130 31337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:29:28.264348 31338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:28.266317 31338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:28.276652 31338 catalog_manager.cc:1347] Generated new cluster ID: e785e600e15344a89e010793c6639502
I20260925 01:29:28.276789 31338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:28.293526 31338 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:28.294956 31338 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:28.301441 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:28.291285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.27.160.62:46381)
0925 01:29:28.291479 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:28.291486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:28.292191 (+ 705us) server_negotiation.cc:407] Connection header received
0925 01:29:28.292471 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:28.292477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:28.292575 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:28.292711 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:28.294109 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:28.295411 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:28.297267 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:28.298065 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:28.298223 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:28.299464 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:28.299472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:28.299477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:28.300416 (+ 939us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:28.300508 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:28.300556 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:28.300683 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:28.300897 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 01:29:28.301035 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:29:28.302838 31338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc4406b96a79419783a2f98a4cba0aea: Generated new TSK 0
I20260925 01:29:28.303687 31338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:29:28.311888 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31250
I20260925 01:29:28.313541 31312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:28.403967 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 31250
2026-09-25T01:29:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1967 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:28 dist-test-slave-h6hf krb5kdc[31351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:28 dist-test-slave-h6hf krb5kdc[31351](info): set up 2 sockets
Sep 25 01:29:28 dist-test-slave-h6hf krb5kdc[31351](info): commencing operation
krb5kdc: starting...
W20260925 01:29:29.506446 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:29 dist-test-slave-h6hf krb5kdc[31351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:29.708184 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:42945
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:42945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:30.028544 31364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:30.028937 31364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:30.029157 31364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:30.037789 31364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:30.037909 31364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:30.037956 31364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:30.038072 31364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:30.038142 31364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:30.048117 31364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:42945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:30.049507 31364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:30.050956 31364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:30.062947 31372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:30.062917 31370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:30.062980 31369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:30.063755 31364 server_base.cc:1034] running on GCE node
I20260925 01:29:30.064564 31364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:30.066105 31364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:30.067305 31364 hybrid_clock.cc:648] HybridClock initialized: now 1790299770067257 us; error 47 us; skew 500 ppm
I20260925 01:29:30.067653 31364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:30 dist-test-slave-h6hf krb5kdc[31351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299770, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:30.070703 31364 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:30.072171 31364 webserver.cc:460] Webserver started at http://127.27.160.62:40387/ using document root <none> and password file <none>
I20260925 01:29:30.072867 31364 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:30.073011 31364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:30.073371 31364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:30.076309 31364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "4b4eba26df834876b74f7839a843b948"
format_stamp: "Formatted at 2026-09-25 01:29:30 on dist-test-slave-h6hf"
server_key: "6fe72a980c7a7eb1bb1a2744cda4ea4c"
server_key_iv: "2fb157d5a3fb672c483c4e41b09dadff"
server_key_version: "encryptionkey@0"
I20260925 01:29:30.077086 31364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "4b4eba26df834876b74f7839a843b948"
format_stamp: "Formatted at 2026-09-25 01:29:30 on dist-test-slave-h6hf"
server_key: "6fe72a980c7a7eb1bb1a2744cda4ea4c"
server_key_iv: "2fb157d5a3fb672c483c4e41b09dadff"
server_key_version: "encryptionkey@0"
I20260925 01:29:30.082150 31364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:29:30.085469 31379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:30.086742 31364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:29:30.086927 31364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "4b4eba26df834876b74f7839a843b948"
format_stamp: "Formatted at 2026-09-25 01:29:30 on dist-test-slave-h6hf"
server_key: "6fe72a980c7a7eb1bb1a2744cda4ea4c"
server_key_iv: "2fb157d5a3fb672c483c4e41b09dadff"
server_key_version: "encryptionkey@0"
I20260925 01:29:30.087121 31364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:30.100760 31364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:30.105049 31364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:30.105386 31364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:30.124157 31364 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:42945
I20260925 01:29:30.124178 31430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:42945 every 8 connection(s)
I20260925 01:29:30.125649 31364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:30.127815 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 31364
I20260925 01:29:30.128029 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:30.128675 28288 external_mini_cluster.cc:1428] Setting key 45cd00b22650549b91300d6ee78ec066
I20260925 01:29:30.129622 31431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:29:30 dist-test-slave-h6hf krb5kdc[31351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:30.143499 31431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948: Bootstrap starting.
I20260925 01:29:30.146984 31431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:30.147995 31431 log.cc:824] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:30.150986 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:30.130801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.27.160.62:42945)
0925 01:29:30.131577 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:30.131592 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:30.131650 (+ 58us) server_negotiation.cc:407] Connection header received
0925 01:29:30.133297 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:30.133326 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:30.133856 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:30.134316 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:30.135512 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:30.137568 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:30.138771 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:30.139807 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:30.140012 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:30.143810 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:30.143819 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:30.143833 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:30.143869 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:30.146888 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:30.147660 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:30.147668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:30.147674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:30.147836 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:30.148239 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:30.148244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:30.148247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:30.148783 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:30.149167 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:30.149541 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 01:29:30.150026 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":318,"threads_started":1}
I20260925 01:29:30.151062 31431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948: No bootstrap required, opened a new log
I20260925 01:29:30.158604 31431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4eba26df834876b74f7839a843b948" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42945 } }
I20260925 01:29:30.159010 31431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:30.159121 31431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4eba26df834876b74f7839a843b948, State: Initialized, Role: FOLLOWER
I20260925 01:29:30.159663 31431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4eba26df834876b74f7839a843b948" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42945 } }
I20260925 01:29:30.159899 31431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:30.160043 31431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:30.160218 31431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:30.162423 31431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4eba26df834876b74f7839a843b948" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42945 } }
I20260925 01:29:30.162972 31431 leader_election.cc:302] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b4eba26df834876b74f7839a843b948; no voters:
I20260925 01:29:30.163561 31431 leader_election.cc:288] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:30.163700 31436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:30.164402 31436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [term 1 LEADER]: Becoming Leader. State: Replica: 4b4eba26df834876b74f7839a843b948, State: Running, Role: LEADER
I20260925 01:29:30.164894 31436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4eba26df834876b74f7839a843b948" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42945 } }
I20260925 01:29:30.165702 31431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:30.167753 31437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b4eba26df834876b74f7839a843b948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4eba26df834876b74f7839a843b948" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42945 } } }
I20260925 01:29:30.167686 31438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b4eba26df834876b74f7839a843b948. Latest consensus state: current_term: 1 leader_uuid: "4b4eba26df834876b74f7839a843b948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4eba26df834876b74f7839a843b948" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42945 } } }
I20260925 01:29:30.169546 31438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:30.170504 31437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:30.171511 31445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:30.176483 31445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:30.184778 31445 catalog_manager.cc:1347] Generated new cluster ID: 725b025c794640abad7bd971d8371c90
I20260925 01:29:30.184893 31445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:30.213814 31445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:30.215736 31445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:30.222189 31445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b4eba26df834876b74f7839a843b948: Generated new TSK 0
I20260925 01:29:30.223275 31445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:29:30.268513 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42945
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:30.616829 31457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:30.617177 31457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:30.617303 31457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:30.624377 31457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:30.624490 31457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:30.624601 31457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:30.634423 31457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:30.635733 31457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:30.636987 31457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:30.650175 31463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:30.650532 31465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:30.650152 31462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:30.651825 31457 server_base.cc:1034] running on GCE node
I20260925 01:29:30.652594 31457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:30.654290 31457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:30.655623 31457 hybrid_clock.cc:648] HybridClock initialized: now 1790299770655563 us; error 43 us; skew 500 ppm
I20260925 01:29:30.656049 31457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:30 dist-test-slave-h6hf krb5kdc[31351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299770, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:30.663136 31457 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:29:30.664755 31457 webserver.cc:460] Webserver started at http://127.27.160.1:34039/ using document root <none> and password file <none>
I20260925 01:29:30.665537 31457 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:30.665721 31457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:30.666029 31457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:30.669086 31457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "95b5eebf0aa14f3e8f6c15267955e5c7"
format_stamp: "Formatted at 2026-09-25 01:29:30 on dist-test-slave-h6hf"
server_key: "a77324b4dae6f3c0a5702b731f8b7508"
server_key_iv: "827a8651906694eabcfbc9864f1b1b8b"
server_key_version: "encryptionkey@0"
I20260925 01:29:30.669924 31457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "95b5eebf0aa14f3e8f6c15267955e5c7"
format_stamp: "Formatted at 2026-09-25 01:29:30 on dist-test-slave-h6hf"
server_key: "a77324b4dae6f3c0a5702b731f8b7508"
server_key_iv: "827a8651906694eabcfbc9864f1b1b8b"
server_key_version: "encryptionkey@0"
I20260925 01:29:30.675554 31457 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:29:30.679086 31472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:30.680552 31457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:29:30.680687 31457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "95b5eebf0aa14f3e8f6c15267955e5c7"
format_stamp: "Formatted at 2026-09-25 01:29:30 on dist-test-slave-h6hf"
server_key: "a77324b4dae6f3c0a5702b731f8b7508"
server_key_iv: "827a8651906694eabcfbc9864f1b1b8b"
server_key_version: "encryptionkey@0"
I20260925 01:29:30.680856 31457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:30.703577 31457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:30.707743 31457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:30.707991 31457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:30.708966 31457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:30.710333 31457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:30.710407 31457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:30.710554 31457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:30.710618 31457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:30.744594 31457 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:46463
I20260925 01:29:30.744704 31584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:46463 every 8 connection(s)
I20260925 01:29:30.746038 31457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:30.749293 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 31457
I20260925 01:29:30.749506 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:30.750401 28288 external_mini_cluster.cc:1428] Setting key 8d590e9ef0ccd9ea8f5a015935a15f22
Sep 25 01:29:30 dist-test-slave-h6hf krb5kdc[31351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299770, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:30.770278 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:30.748607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:36977 (local address 127.27.160.62:42945)
0925 01:29:30.749070 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:30.749077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:30.750415 (+ 1338us) server_negotiation.cc:407] Connection header received
0925 01:29:30.752651 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:30.752656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:30.752716 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:30.752806 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:30.755005 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:30.755715 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:30.758253 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:30.758905 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:30.759038 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:30.762880 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:30.762889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:30.762897 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:30.762944 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:30.766565 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:30.767486 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:30.767495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:30.767501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:30.767659 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:30.768124 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:30.768132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:30.768138 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:30.768553 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:30.768724 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:30.769574 (+ 850us) server_negotiation.cc:299] Negotiation successful
0925 01:29:30.769838 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":197,"threads_started":1}
I20260925 01:29:30.771137 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:30.749301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:42945 (local address 127.27.160.1:36977)
0925 01:29:30.750208 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:30.750262 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:30.752345 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:30.753106 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:30.753144 (+ 38us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:30.753960 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:30.754816 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:30.754845 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:30.755876 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:30.755885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:30.758105 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:30.758123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:30.759172 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:30.759176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:30.759389 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:30.760092 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:30.760113 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:30.762625 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:30.766696 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:30.766702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:30.766716 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:30.767350 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:30.767773 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:30.767775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:30.767778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:30.767995 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:30.768701 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:30.768706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:30.769085 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 01:29:30.769393 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 01:29:30.769803 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":330,"threads_started":1}
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:29:30.772864 31585 heartbeater.cc:347] Connected to a master server at 127.27.160.62:42945
I20260925 01:29:30.773267 31585 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:30.774127 31585 heartbeater.cc:511] Master 127.27.160.62:42945 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
I20260925 01:29:30.776515 31396 ts_manager.cc:194] Registered new tserver with Master: 95b5eebf0aa14f3e8f6c15267955e5c7 (127.27.160.1:46463)
I20260925 01:29:30.778249 31396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:36977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:30.800019 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42945
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:31.122172 31591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:31.122473 31591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:31.122642 31591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:31.130658 31591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:31.130767 31591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:31.130944 31591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:31.140756 31591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:31.142082 31591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:31.143385 31591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:31.156606 31596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:31.156690 31599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:31.156663 31597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:31.158027 31591 server_base.cc:1034] running on GCE node
I20260925 01:29:31.158630 31591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:31.159550 31591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:31.160781 31591 hybrid_clock.cc:648] HybridClock initialized: now 1790299771160741 us; error 47 us; skew 500 ppm
I20260925 01:29:31.161100 31591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:31.164233 31591 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:31.165838 31591 webserver.cc:460] Webserver started at http://127.27.160.2:37297/ using document root <none> and password file <none>
I20260925 01:29:31.166515 31591 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:31.166626 31591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:31.166869 31591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:31.169796 31591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "8f12785e757549caa0ac38e16fd8c46a"
format_stamp: "Formatted at 2026-09-25 01:29:31 on dist-test-slave-h6hf"
server_key: "3c5a81552fc95bae7065fd6a4796bc66"
server_key_iv: "8379af2b87d65f1e4188b422e5170b1e"
server_key_version: "encryptionkey@0"
I20260925 01:29:31.170516 31591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "8f12785e757549caa0ac38e16fd8c46a"
format_stamp: "Formatted at 2026-09-25 01:29:31 on dist-test-slave-h6hf"
server_key: "3c5a81552fc95bae7065fd6a4796bc66"
server_key_iv: "8379af2b87d65f1e4188b422e5170b1e"
server_key_version: "encryptionkey@0"
I20260925 01:29:31.175825 31591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:29:31.179337 31606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:31.180713 31591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:29:31.180847 31591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "8f12785e757549caa0ac38e16fd8c46a"
format_stamp: "Formatted at 2026-09-25 01:29:31 on dist-test-slave-h6hf"
server_key: "3c5a81552fc95bae7065fd6a4796bc66"
server_key_iv: "8379af2b87d65f1e4188b422e5170b1e"
server_key_version: "encryptionkey@0"
I20260925 01:29:31.181002 31591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:31.202709 31591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:31.206770 31591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:31.207021 31591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:31.207958 31591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:31.209267 31591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:31.209342 31591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:31.209424 31591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:31.209467 31591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:31.242926 31591 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:36865
I20260925 01:29:31.243017 31718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:36865 every 8 connection(s)
I20260925 01:29:31.244241 31591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:31.248817 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 31591
I20260925 01:29:31.249017 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:31.249492 28288 external_mini_cluster.cc:1428] Setting key 1670ab7f05e371845a4fd7406dbc964c
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:31.265494 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:31.246859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:44143 (local address 127.27.160.62:42945)
0925 01:29:31.247050 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:31.247055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:31.248263 (+ 1208us) server_negotiation.cc:407] Connection header received
0925 01:29:31.250250 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:31.250256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:31.250343 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:31.250442 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:31.251982 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.253228 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.254837 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.255592 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.255739 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:31.259782 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:31.259787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:31.259790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:31.259822 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:31.262377 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.263114 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.263121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.263124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.263260 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.263688 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.263694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.263697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.263961 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:31.264121 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:31.264859 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 01:29:31.265092 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:29:31.266597 31721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:31.247351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:42945 (local address 127.27.160.2:44143)
0925 01:29:31.248102 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:31.248140 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:31.249964 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:31.250632 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:31.250640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:31.251241 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:31.251798 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:31.251819 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.253436 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.253460 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:31.254687 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:31.254699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.255802 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.255808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:31.256003 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:31.256748 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:31.256782 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:31.259550 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:31.262526 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:31.262535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:31.262553 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:31.262979 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:31.263375 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:31.263380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:31.263383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:31.263577 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:31.264131 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:31.264142 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:31.264492 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 01:29:31.264773 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 01:29:31.265394 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":324,"threads_started":1}
I20260925 01:29:31.268173 31719 heartbeater.cc:347] Connected to a master server at 127.27.160.62:42945
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:29:31.268481 31719 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:31.269290 31719 heartbeater.cc:511] Master 127.27.160.62:42945 requested a full tablet report, sending...
I20260925 01:29:31.270886 31396 ts_manager.cc:194] Registered new tserver with Master: 8f12785e757549caa0ac38e16fd8c46a (127.27.160.2:36865)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
I20260925 01:29:31.271935 31396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:44143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:31.296440 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42945
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:31.628187 31724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:31.628472 31724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:31.628634 31724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:31.636044 31724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:31.636163 31724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:31.636340 31724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:31.646679 31724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:31.648052 31724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:31.649510 31724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:31.662963 31730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:31.663010 31732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:31.662917 31729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:31.663640 31724 server_base.cc:1034] running on GCE node
I20260925 01:29:31.664386 31724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:31.665478 31724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:31.666718 31724 hybrid_clock.cc:648] HybridClock initialized: now 1790299771666642 us; error 57 us; skew 500 ppm
I20260925 01:29:31.667099 31724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:31.670569 31724 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:31.672242 31724 webserver.cc:460] Webserver started at http://127.27.160.3:34393/ using document root <none> and password file <none>
I20260925 01:29:31.672958 31724 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:31.673086 31724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:31.673444 31724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:31.676501 31724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "dc01c8eab2ac4daaa0b8f57f22b5d133"
format_stamp: "Formatted at 2026-09-25 01:29:31 on dist-test-slave-h6hf"
server_key: "a7608c98de860224ac12d97c0477574e"
server_key_iv: "089059986bcbb502e3389b12dc53cc6f"
server_key_version: "encryptionkey@0"
I20260925 01:29:31.677467 31724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "dc01c8eab2ac4daaa0b8f57f22b5d133"
format_stamp: "Formatted at 2026-09-25 01:29:31 on dist-test-slave-h6hf"
server_key: "a7608c98de860224ac12d97c0477574e"
server_key_iv: "089059986bcbb502e3389b12dc53cc6f"
server_key_version: "encryptionkey@0"
I20260925 01:29:31.682773 31724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:29:31.686648 31739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:31.688082 31724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:29:31.688273 31724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "dc01c8eab2ac4daaa0b8f57f22b5d133"
format_stamp: "Formatted at 2026-09-25 01:29:31 on dist-test-slave-h6hf"
server_key: "a7608c98de860224ac12d97c0477574e"
server_key_iv: "089059986bcbb502e3389b12dc53cc6f"
server_key_version: "encryptionkey@0"
I20260925 01:29:31.688483 31724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:31.708935 31724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:31.713656 31724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:31.713989 31724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:31.715025 31724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:31.716446 31724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:31.716524 31724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:31.716706 31724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:31.716759 31724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:31.750871 31724 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:43769
I20260925 01:29:31.750932 31851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:43769 every 8 connection(s)
I20260925 01:29:31.752307 31724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:31.756659 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 31724
I20260925 01:29:31.756837 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:31.757365 28288 external_mini_cluster.cc:1428] Setting key 8d4aa6b2f4ac280e8638f3562e5d7d64
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:31.778373 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:31.754886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:43705 (local address 127.27.160.62:42945)
0925 01:29:31.755082 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:31.755087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:31.756328 (+ 1241us) server_negotiation.cc:407] Connection header received
0925 01:29:31.758446 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:31.758452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:31.758572 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:31.758682 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:31.760168 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.760920 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.762509 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.762965 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.763106 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:31.772521 (+ 9415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:31.772527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:31.772533 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:31.772577 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:31.775321 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.776038 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.776042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.776044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.776168 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.776590 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.776594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.776597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.776899 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:31.777084 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:31.777868 (+ 784us) server_negotiation.cc:299] Negotiation successful
0925 01:29:31.778100 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:29:31.779229 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:31.755379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:42945 (local address 127.27.160.3:43705)
0925 01:29:31.756158 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:31.756202 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:31.758119 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:31.758940 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:31.758948 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:31.759445 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:31.760005 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:31.760022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.761081 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.761085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:31.762345 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:31.762357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.763196 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.763204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:31.763444 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:31.769673 (+ 6229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:31.769702 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:31.772305 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:31.775453 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:31.775460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:31.775473 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:31.775916 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:31.776262 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:31.776266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:31.776268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:31.776442 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:31.777080 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:31.777086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:31.777473 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 01:29:31.777730 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 01:29:31.778127 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":274,"threads_started":1}
I20260925 01:29:31.780843 31852 heartbeater.cc:347] Connected to a master server at 127.27.160.62:42945
I20260925 01:29:31.781196 31852 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:31.781415 31585 heartbeater.cc:503] Master 127.27.160.62:42945 was elected leader, sending a full tablet report...
I20260925 01:29:31.782136 31852 heartbeater.cc:511] Master 127.27.160.62:42945 requested a full tablet report, sending...
I20260925 01:29:31.783617 31396 ts_manager.cc:194] Registered new tserver with Master: dc01c8eab2ac4daaa0b8f57f22b5d133 (127.27.160.3:43769)
I20260925 01:29:31.784729 31396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:43705
I20260925 01:29:31.785195 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:31.836822 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:31.821898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.27.160.62:42945)
0925 01:29:31.822178 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:31.822186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:31.822549 (+ 363us) server_negotiation.cc:407] Connection header received
0925 01:29:31.822794 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:31.822800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:31.822870 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:31.823075 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:31.824564 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.825433 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.826782 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.827225 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.827373 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:31.831419 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:31.831425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:31.831428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:31.831465 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:31.834026 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.834717 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.834730 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.834734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.834869 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.835288 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.835294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.835297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.835648 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:31.835929 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:31.836242 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 01:29:31.836471 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260925 01:29:31.837522 31396 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44268
I20260925 01:29:31.839591 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31457
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:31.855684 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:31.840625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.27.160.1:46463)
0925 01:29:31.841576 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:31.841583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:31.841609 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:29:31.841664 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:31.841671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:31.841876 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:31.842042 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:31.843249 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.844237 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.845498 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.846072 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.846226 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:31.850131 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:31.850143 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:31.850157 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:31.850189 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:31.852811 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.853559 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.853567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.853570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.853708 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.854091 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.854096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.854099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.854443 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:31.854761 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:31.855062 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 01:29:31.855330 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":646,"threads_started":1}
W20260925 01:29:31.857048 31580 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57966
W20260925 01:29:31.858369 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31457: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:31.858606 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 31457
I20260925 01:29:31.876073 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31591
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:29:31.892302 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:31.876928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50176 (local address 127.27.160.2:36865)
0925 01:29:31.877600 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:31.877608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:31.877636 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:29:31.877701 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:31.877708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:31.877939 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:31.878083 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:31.879288 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.880383 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.881740 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.882326 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.882434 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:31.886577 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:31.886589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:31.886605 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:31.886646 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:31.889558 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.890174 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.890181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.890184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.890309 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.890643 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.890652 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.890655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.891055 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:31.891290 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:31.891609 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 01:29:31.891850 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":354,"threads_started":1}
W20260925 01:29:31.893782 31714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50176
W20260925 01:29:31.894908 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:31.895067 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 31591
I20260925 01:29:31.910471 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31724
Sep 25 01:29:31 dist-test-slave-h6hf krb5kdc[31351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299771, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:29:31.927176 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:31.911468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.27.160.3:43769)
0925 01:29:31.912141 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:31.912148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:31.912177 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:29:31.912237 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:31.912244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:31.912486 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:31.912622 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:31.914025 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.915191 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.916382 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:31.916983 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:31.917086 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:31.921435 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:31.921467 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:31.921483 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:31.921523 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:31.924571 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.925233 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.925246 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.925253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.925391 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:31.925737 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:31.925740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:31.925742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:31.926091 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:31.926321 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:31.926546 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 01:29:31.926746 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":358,"threads_started":1}
W20260925 01:29:31.928514 31847 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49416
W20260925 01:29:31.929778 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:29:31.929965 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 31724
I20260925 01:29:31.945453 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31364
I20260925 01:29:31.946658 31426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:32.066469 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 31364
2026-09-25T01:29:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3668 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:32 dist-test-slave-h6hf krb5kdc[31869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:32 dist-test-slave-h6hf krb5kdc[31869](info): set up 2 sockets
Sep 25 01:29:32 dist-test-slave-h6hf krb5kdc[31869](info): commencing operation
krb5kdc: starting...
W20260925 01:29:33.194501 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:33 dist-test-slave-h6hf krb5kdc[31869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:33.408828 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:46471
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:46471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:33.741242 31882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:33.741559 31882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:33.741729 31882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:33.749110 31882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:33.749246 31882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:33.749356 31882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:33.749457 31882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:33.749547 31882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:33.759924 31882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:46471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:33.761358 31882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:33.762729 31882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:33.774703 31887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:33.774725 31890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:33.774755 31888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:33.775686 31882 server_base.cc:1034] running on GCE node
I20260925 01:29:33.776476 31882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:33.778092 31882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:33.779306 31882 hybrid_clock.cc:648] HybridClock initialized: now 1790299773779260 us; error 33 us; skew 500 ppm
I20260925 01:29:33.779695 31882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:33 dist-test-slave-h6hf krb5kdc[31869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299773, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:33.782959 31882 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:33.784495 31882 webserver.cc:460] Webserver started at http://127.27.160.62:35095/ using document root <none> and password file <none>
I20260925 01:29:33.785295 31882 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:33.785405 31882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:33.785724 31882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:33.788753 31882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "d0b4bb8fa3f1423fabcca839a0b9263b"
format_stamp: "Formatted at 2026-09-25 01:29:33 on dist-test-slave-h6hf"
server_key: "52cf52f1833cbc0dce375bacb9547253"
server_key_iv: "0a39528fe47ceff001d64a4c5ef0ac41"
server_key_version: "encryptionkey@0"
I20260925 01:29:33.789585 31882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "d0b4bb8fa3f1423fabcca839a0b9263b"
format_stamp: "Formatted at 2026-09-25 01:29:33 on dist-test-slave-h6hf"
server_key: "52cf52f1833cbc0dce375bacb9547253"
server_key_iv: "0a39528fe47ceff001d64a4c5ef0ac41"
server_key_version: "encryptionkey@0"
I20260925 01:29:33.795003 31882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:29:33.798642 31897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:33.800071 31882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 01:29:33.800261 31882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "d0b4bb8fa3f1423fabcca839a0b9263b"
format_stamp: "Formatted at 2026-09-25 01:29:33 on dist-test-slave-h6hf"
server_key: "52cf52f1833cbc0dce375bacb9547253"
server_key_iv: "0a39528fe47ceff001d64a4c5ef0ac41"
server_key_version: "encryptionkey@0"
I20260925 01:29:33.800475 31882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:33.845217 31882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:33.849592 31882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:33.849900 31882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:33.869863 31882 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:46471
I20260925 01:29:33.869881 31948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:46471 every 8 connection(s)
I20260925 01:29:33.871331 31882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:33.875234 31949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:33.879156 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 31882
I20260925 01:29:33.879400 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:33.880019 28288 external_mini_cluster.cc:1428] Setting key 78e578dba9169627e41d7186937e5879
I20260925 01:29:33.887835 31949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b: Bootstrap starting.
I20260925 01:29:33.891624 31949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:33.892958 31949 log.cc:824] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:33.896057 31949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b: No bootstrap required, opened a new log
Sep 25 01:29:33 dist-test-slave-h6hf krb5kdc[31869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:33.906327 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:33.882249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.27.160.62:46471)
0925 01:29:33.883104 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:33.883120 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:33.883171 (+ 51us) server_negotiation.cc:407] Connection header received
0925 01:29:33.886014 (+ 2843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:33.886048 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:33.886594 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:33.887057 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:33.888559 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:33.890673 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:33.892027 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:33.893105 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:33.893305 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:33.897328 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:33.897339 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:33.897358 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:33.897402 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:33.901501 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:33.902213 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:33.902223 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:33.902232 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:33.902445 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:33.902782 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:33.902787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:33.902791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:33.903552 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:33.903830 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:33.904315 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 01:29:33.904795 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":355,"threads_started":1}
I20260925 01:29:33.907220 31949 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b4bb8fa3f1423fabcca839a0b9263b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46471 } }
I20260925 01:29:33.907568 31949 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:33.907743 31949 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b4bb8fa3f1423fabcca839a0b9263b, State: Initialized, Role: FOLLOWER
I20260925 01:29:33.908420 31949 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b4bb8fa3f1423fabcca839a0b9263b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46471 } }
I20260925 01:29:33.908677 31949 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:33.908808 31949 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:33.909001 31949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:33.911273 31949 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b4bb8fa3f1423fabcca839a0b9263b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46471 } }
I20260925 01:29:33.911809 31949 leader_election.cc:302] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0b4bb8fa3f1423fabcca839a0b9263b; no voters:
I20260925 01:29:33.912431 31949 leader_election.cc:288] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:33.912590 31954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:33.914099 31954 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [term 1 LEADER]: Becoming Leader. State: Replica: d0b4bb8fa3f1423fabcca839a0b9263b, State: Running, Role: LEADER
I20260925 01:29:33.914610 31954 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b4bb8fa3f1423fabcca839a0b9263b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46471 } }
I20260925 01:29:33.914852 31949 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:33.917740 31956 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b4bb8fa3f1423fabcca839a0b9263b. Latest consensus state: current_term: 1 leader_uuid: "d0b4bb8fa3f1423fabcca839a0b9263b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b4bb8fa3f1423fabcca839a0b9263b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46471 } } }
I20260925 01:29:33.918207 31956 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:33.917766 31955 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0b4bb8fa3f1423fabcca839a0b9263b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b4bb8fa3f1423fabcca839a0b9263b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 46471 } } }
I20260925 01:29:33.918462 31955 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:33.919580 31962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:33.924582 31962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:33.934242 31962 catalog_manager.cc:1347] Generated new cluster ID: 1992516859984228b67932356d64253e
I20260925 01:29:33.934377 31962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:33.947737 31962 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:33.949065 31962 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:33.956297 31962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0b4bb8fa3f1423fabcca839a0b9263b: Generated new TSK 0
I20260925 01:29:33.957223 31962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:29:34.009634 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:46471
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:34.345481 31975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:34.345781 31975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:34.345875 31975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:34.353415 31975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:34.353533 31975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:34.353670 31975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:34.363848 31975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:34.365142 31975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:34.366415 31975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:34.380411 31980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:34.380509 31983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:34.380407 31981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:34.381395 31975 server_base.cc:1034] running on GCE node
I20260925 01:29:34.382074 31975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:34.383065 31975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:34.384356 31975 hybrid_clock.cc:648] HybridClock initialized: now 1790299774384306 us; error 59 us; skew 500 ppm
I20260925 01:29:34.384763 31975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:34 dist-test-slave-h6hf krb5kdc[31869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299774, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:34.388715 31975 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:29:34.390547 31975 webserver.cc:460] Webserver started at http://127.27.160.1:33931/ using document root <none> and password file <none>
I20260925 01:29:34.391350 31975 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:34.391522 31975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:34.391888 31975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:34.395259 31975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "546d8074ab0f439492c3ee5d8db5507a"
format_stamp: "Formatted at 2026-09-25 01:29:34 on dist-test-slave-h6hf"
server_key: "69ba31d0e2460f686c39cbe02619f2c8"
server_key_iv: "c1e61f85830f2b3713ca1525f5b8de52"
server_key_version: "encryptionkey@0"
I20260925 01:29:34.396158 31975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "546d8074ab0f439492c3ee5d8db5507a"
format_stamp: "Formatted at 2026-09-25 01:29:34 on dist-test-slave-h6hf"
server_key: "69ba31d0e2460f686c39cbe02619f2c8"
server_key_iv: "c1e61f85830f2b3713ca1525f5b8de52"
server_key_version: "encryptionkey@0"
I20260925 01:29:34.402102 31975 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 01:29:34.406270 31990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:34.407838 31975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260925 01:29:34.408054 31975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "546d8074ab0f439492c3ee5d8db5507a"
format_stamp: "Formatted at 2026-09-25 01:29:34 on dist-test-slave-h6hf"
server_key: "69ba31d0e2460f686c39cbe02619f2c8"
server_key_iv: "c1e61f85830f2b3713ca1525f5b8de52"
server_key_version: "encryptionkey@0"
I20260925 01:29:34.408309 31975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:34.435426 31975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:34.440188 31975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:34.440495 31975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:34.441610 31975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:34.443002 31975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:34.443109 31975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:34.443252 31975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:34.443324 31975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:34.478796 31975 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:41135
I20260925 01:29:34.478847 32102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:41135 every 8 connection(s)
I20260925 01:29:34.480127 31975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:34.480765 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 31975
I20260925 01:29:34.481000 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:34.481848 28288 external_mini_cluster.cc:1428] Setting key 43901bfac86c25424613e1ca0c33d8e2
Sep 25 01:29:34 dist-test-slave-h6hf krb5kdc[31869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299774, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:29:34.503455 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:34.482998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:34063 (local address 127.27.160.62:46471)
0925 01:29:34.484154 (+ 1156us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:34.484160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:34.484402 (+ 242us) server_negotiation.cc:407] Connection header received
0925 01:29:34.486421 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:34.486429 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:34.486534 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:34.486657 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:34.488232 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:34.489443 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:34.491112 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:34.491877 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:34.492036 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:34.495928 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:34.495946 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:34.495954 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:34.496004 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:34.500011 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:34.500824 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:34.500831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:34.500835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:34.501085 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:34.501554 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:34.501559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:34.501564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:34.501962 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:34.502130 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:34.502850 (+ 720us) server_negotiation.cc:299] Negotiation successful
0925 01:29:34.503105 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044,"thread_start_us":956,"threads_started":1}
I20260925 01:29:34.504583 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:34.483363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:46471 (local address 127.27.160.1:34063)
0925 01:29:34.484228 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:34.484277 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:34.486121 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:34.486879 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:34.486887 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:34.487416 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:34.488041 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:34.488060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:34.489602 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:34.489605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:34.490938 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:34.490950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:34.492118 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:34.492123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:34.492326 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:34.493175 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:34.493209 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:34.495675 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:34.500193 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:34.500201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:34.500217 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:34.500683 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:34.501247 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:34.501250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:34.501252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:34.501435 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:34.502100 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:34.502106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:34.502446 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 01:29:34.502736 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 01:29:34.503299 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
I20260925 01:29:34.506404 32103 heartbeater.cc:347] Connected to a master server at 127.27.160.62:46471
I20260925 01:29:34.506757 32103 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:34.507691 32103 heartbeater.cc:511] Master 127.27.160.62:46471 requested a full tablet report, sending...
I20260925 01:29:34.510193 31914 ts_manager.cc:194] Registered new tserver with Master: 546d8074ab0f439492c3ee5d8db5507a (127.27.160.1:41135)
I20260925 01:29:34.511970 31914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:34063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:34.532867 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:46471
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:34.869398 32109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:34.869711 32109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:34.869891 32109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:34.877558 32109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:34.877677 32109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:34.877869 32109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:34.887915 32109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:34.889335 32109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:34.890731 32109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:34.904695 32114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:34.904753 32117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:34.904635 32115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:34.906158 32109 server_base.cc:1034] running on GCE node
I20260925 01:29:34.906826 32109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:34.907829 32109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:34.909073 32109 hybrid_clock.cc:648] HybridClock initialized: now 1790299774909040 us; error 33 us; skew 500 ppm
I20260925 01:29:34.909503 32109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:34 dist-test-slave-h6hf krb5kdc[31869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299774, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:34.912772 32109 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:34.914438 32109 webserver.cc:460] Webserver started at http://127.27.160.2:43909/ using document root <none> and password file <none>
I20260925 01:29:34.915274 32109 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:34.915409 32109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:34.915746 32109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:34.919013 32109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "6010cdcba10443b7a5672144adfb3802"
format_stamp: "Formatted at 2026-09-25 01:29:34 on dist-test-slave-h6hf"
server_key: "c6d9afa8f59b42c55ccb70a1486c8921"
server_key_iv: "f32f4ad27f7083f608085b8f530183de"
server_key_version: "encryptionkey@0"
I20260925 01:29:34.919863 32109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "6010cdcba10443b7a5672144adfb3802"
format_stamp: "Formatted at 2026-09-25 01:29:34 on dist-test-slave-h6hf"
server_key: "c6d9afa8f59b42c55ccb70a1486c8921"
server_key_iv: "f32f4ad27f7083f608085b8f530183de"
server_key_version: "encryptionkey@0"
I20260925 01:29:34.925217 32109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 01:29:34.928719 32124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:34.930083 32109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:34.930254 32109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "6010cdcba10443b7a5672144adfb3802"
format_stamp: "Formatted at 2026-09-25 01:29:34 on dist-test-slave-h6hf"
server_key: "c6d9afa8f59b42c55ccb70a1486c8921"
server_key_iv: "f32f4ad27f7083f608085b8f530183de"
server_key_version: "encryptionkey@0"
I20260925 01:29:34.930490 32109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:34.944087 32109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:34.948181 32109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:34.948495 32109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:34.949564 32109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:34.950901 32109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:34.950984 32109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:34.951145 32109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:34.951207 32109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:34.984172 32109 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:38643
I20260925 01:29:34.984216 32236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:38643 every 8 connection(s)
I20260925 01:29:34.985549 32109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:34.992772 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 32109
I20260925 01:29:34.992964 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:34.993430 28288 external_mini_cluster.cc:1428] Setting key ecf38582dfb168ef76e15a8b6246a30b
Sep 25 01:29:34 dist-test-slave-h6hf krb5kdc[31869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299774, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:35.006822 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:34.988282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:55729 (local address 127.27.160.62:46471)
0925 01:29:34.988474 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:34.988478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:34.989676 (+ 1198us) server_negotiation.cc:407] Connection header received
0925 01:29:34.991414 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:34.991419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:34.991486 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:34.991575 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:34.993178 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:34.994009 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:34.995991 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:34.996396 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:34.996503 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:35.000805 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:35.000809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:35.000813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:35.000846 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:35.003482 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.004255 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.004262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.004266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.004413 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.004876 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.004882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.004885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.005254 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:35.005456 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:35.006211 (+ 755us) server_negotiation.cc:299] Negotiation successful
0925 01:29:35.006470 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:29:35.007913 32239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:34.988768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:46471 (local address 127.27.160.2:55729)
0925 01:29:34.989519 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:34.989554 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:34.991172 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:34.991786 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:34.991794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:34.992333 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:34.992967 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:34.992986 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:34.994189 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:34.994194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:34.995838 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:34.995855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:34.996641 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:34.996651 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:34.996993 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:34.997811 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:34.997844 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:35.000561 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:35.003646 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:35.003656 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:35.003674 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:35.004127 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:35.004555 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:35.004560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:35.004564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:35.004761 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:35.005419 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:35.005427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:35.005833 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 01:29:35.006130 (+ 297us) client_negotiation.cc:240] Negotiation successful
0925 01:29:35.006712 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":325,"threads_started":1}
I20260925 01:29:35.009722 32237 heartbeater.cc:347] Connected to a master server at 127.27.160.62:46471
I20260925 01:29:35.010071 32237 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:35.010844 32237 heartbeater.cc:511] Master 127.27.160.62:46471 requested a full tablet report, sending...
I20260925 01:29:35.012724 31914 ts_manager.cc:194] Registered new tserver with Master: 6010cdcba10443b7a5672144adfb3802 (127.27.160.2:38643)
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:29:35.013721 31914 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:55729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:35.041747 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:46471
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:35.376631 32242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:35.376965 32242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:35.377172 32242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:35.384593 32242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:35.384764 32242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:35.384950 32242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:35.395061 32242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:46471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:35.396449 32242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:35.397859 32242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:35.411571 32247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:35.411489 32248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:35.411548 32250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:35.413174 32242 server_base.cc:1034] running on GCE node
I20260925 01:29:35.413865 32242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:35.414896 32242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:35.416167 32242 hybrid_clock.cc:648] HybridClock initialized: now 1790299775416077 us; error 78 us; skew 500 ppm
I20260925 01:29:35.416638 32242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:35 dist-test-slave-h6hf krb5kdc[31869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299775, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:35.420447 32242 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:35.422437 32242 webserver.cc:460] Webserver started at http://127.27.160.3:38897/ using document root <none> and password file <none>
I20260925 01:29:35.423213 32242 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:35.423369 32242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:35.423707 32242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:35.426894 32242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "a7551fd813134e5cb91794358cc54fe8"
format_stamp: "Formatted at 2026-09-25 01:29:35 on dist-test-slave-h6hf"
server_key: "5a51d60be246cadc354e592d7faeb42b"
server_key_iv: "0d3f631aaba001b4de478f16124d7aa8"
server_key_version: "encryptionkey@0"
I20260925 01:29:35.427716 32242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "a7551fd813134e5cb91794358cc54fe8"
format_stamp: "Formatted at 2026-09-25 01:29:35 on dist-test-slave-h6hf"
server_key: "5a51d60be246cadc354e592d7faeb42b"
server_key_iv: "0d3f631aaba001b4de478f16124d7aa8"
server_key_version: "encryptionkey@0"
I20260925 01:29:35.433441 32242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:29:35.437321 32257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:35.438771 32242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:35.438970 32242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "a7551fd813134e5cb91794358cc54fe8"
format_stamp: "Formatted at 2026-09-25 01:29:35 on dist-test-slave-h6hf"
server_key: "5a51d60be246cadc354e592d7faeb42b"
server_key_iv: "0d3f631aaba001b4de478f16124d7aa8"
server_key_version: "encryptionkey@0"
I20260925 01:29:35.439203 32242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:35.462327 32242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:35.467103 32242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:35.467425 32242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:35.468459 32242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:35.469969 32242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:35.470084 32242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:35.470238 32242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:35.470311 32242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:35.505224 32242 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:34155
I20260925 01:29:35.505323 32369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:34155 every 8 connection(s)
I20260925 01:29:35.506560 32242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:35.512657 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 32242
I20260925 01:29:35.512957 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:35.513617 28288 external_mini_cluster.cc:1428] Setting key 707bfc21c86ce0f61f64730755849e01
I20260925 01:29:35.515456 32103 heartbeater.cc:503] Master 127.27.160.62:46471 was elected leader, sending a full tablet report...
Sep 25 01:29:35 dist-test-slave-h6hf krb5kdc[31869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299775, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:35.530802 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:35.509759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:45929 (local address 127.27.160.62:46471)
0925 01:29:35.510236 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:35.510240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:35.511345 (+ 1105us) server_negotiation.cc:407] Connection header received
0925 01:29:35.513687 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:35.513694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:35.513799 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:35.513928 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:35.516877 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.517954 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.519831 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.520383 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.520510 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:35.524382 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:35.524387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:35.524390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:35.524425 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:35.527046 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.527829 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.527835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.527837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.527968 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.528417 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.528421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.528423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.528875 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:35.529002 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:35.530139 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0925 01:29:35.530373 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":268,"threads_started":1}
I20260925 01:29:35.531949 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:35.510434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:46471 (local address 127.27.160.3:45929)
0925 01:29:35.511173 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:35.511220 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:35.513316 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:35.514247 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:35.514259 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:35.515109 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:35.516649 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:35.516678 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.518117 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.518122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:35.519609 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:35.519629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.520604 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.520609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:35.520829 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:35.521575 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:35.521599 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:35.524146 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:35.527194 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:35.527201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:35.527217 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:35.527671 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:35.528089 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:35.528094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:35.528098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:35.528319 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:35.529076 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:35.529083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:35.529660 (+ 577us) client_negotiation.cc:769] Sending connection context
0925 01:29:35.530050 (+ 390us) client_negotiation.cc:240] Negotiation successful
0925 01:29:35.530642 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":292,"threads_started":1}
I20260925 01:29:35.533519 32370 heartbeater.cc:347] Connected to a master server at 127.27.160.62:46471
I20260925 01:29:35.533814 32370 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:35.534780 32370 heartbeater.cc:511] Master 127.27.160.62:46471 requested a full tablet report, sending...
I20260925 01:29:35.536732 31913 ts_manager.cc:194] Registered new tserver with Master: a7551fd813134e5cb91794358cc54fe8 (127.27.160.3:34155)
I20260925 01:29:35.538108 31913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:45929
I20260925 01:29:35.543380 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:29:35 dist-test-slave-h6hf krb5kdc[31869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:29:35 dist-test-slave-h6hf krb5kdc[31869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:35.592890 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:35.577261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.27.160.1:41135)
0925 01:29:35.577903 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:35.577911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:35.577940 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:29:35.578118 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:35.578126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:35.578348 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:35.578479 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:35.579894 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.580889 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.582169 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.582735 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.582834 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:35.586924 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:35.586942 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:35.586958 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:35.586997 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:35.590022 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.590661 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.590668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.590670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.590786 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.591196 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.591201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.591204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.591607 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:35.591938 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:35.592168 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 01:29:35.592462 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":314,"threads_started":1}
Sep 25 01:29:35 dist-test-slave-h6hf krb5kdc[31869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:35.612694 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:35.597910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.27.160.62:46471)
0925 01:29:35.598174 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:35.598178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:35.598196 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:29:35.598331 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:35.598335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:35.598410 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:35.598500 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:35.599720 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.600504 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.602042 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.602456 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.602559 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:35.607100 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:35.607148 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:35.607151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:35.607184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:35.609917 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.610565 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.610570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.610572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.610694 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.611111 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.611114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.611116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.611410 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:35.611635 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:35.612068 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 01:29:35.612248 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
W20260925 01:29:35.613400 31913 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40200
I20260925 01:29:35.614460 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31975
I20260925 01:29:35.628574 32378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:35.615600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.27.160.1:41135)
0925 01:29:35.615796 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:35.615801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:35.615841 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:29:35.616034 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:35.616038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:35.616189 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:35.616276 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:35.617534 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.618248 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.619669 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.620205 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.620323 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:35.623162 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:35.623168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:35.623171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:35.623205 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:35.626207 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.626902 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.626908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.626912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.627045 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.627375 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.627378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.627380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.627662 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:35.627843 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:35.628123 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 01:29:35.628293 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:29:35.629300 32098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:35.714036 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 31975
I20260925 01:29:35.730382 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 32109
Sep 25 01:29:35 dist-test-slave-h6hf krb5kdc[31869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:29:35.752976 32383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:35.731283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.27.160.2:38643)
0925 01:29:35.731895 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:35.731902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:35.731929 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:29:35.731990 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:35.731997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:35.732229 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:35.732366 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:35.733811 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.734945 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.736589 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.737252 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.737392 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:35.746707 (+ 9315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:35.746723 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:35.746739 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:35.746782 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:35.750097 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.750814 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.750831 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.750834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.750955 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.751353 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.751358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.751361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.751714 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:35.752025 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:35.752360 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 01:29:35.752626 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":339,"threads_started":1}
I20260925 01:29:35.754305 32232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:35.837918 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 32109
I20260925 01:29:35.855852 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 32242
Sep 25 01:29:35 dist-test-slave-h6hf krb5kdc[31869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:29:35.873762 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:35.856939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.27.160.3:34155)
0925 01:29:35.857662 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:35.857669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:35.857698 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:29:35.857759 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:35.857765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:35.858004 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:35.858206 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:35.859481 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.860510 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.861983 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:35.862575 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:35.862721 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:35.867361 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:35.867373 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:35.867389 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:35.867428 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:35.870829 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.871539 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.871549 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.871553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.871691 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:35.872055 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:35.872060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:35.872063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:35.872465 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:35.872747 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:35.873015 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 01:29:35.873352 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":419,"threads_started":1}
I20260925 01:29:35.875217 32365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:35.958853 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 32242
I20260925 01:29:35.975142 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 31882
I20260925 01:29:35.976431 31944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:36.081843 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 31882
2026-09-25T01:29:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4016 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:36 dist-test-slave-h6hf krb5kdc[32390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:36 dist-test-slave-h6hf krb5kdc[32390](info): set up 2 sockets
Sep 25 01:29:36 dist-test-slave-h6hf krb5kdc[32390](info): commencing operation
krb5kdc: starting...
W20260925 01:29:37.255116 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:37 dist-test-slave-h6hf krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:37.467051 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:43969
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:37.804108 32403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:37.804409 32403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:37.804576 32403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:37.811892 32403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:37.812000 32403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:37.812083 32403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:37.812186 32403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:37.812265 32403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:37.822463 32403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:43969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:37.823840 32403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:37.825265 32403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:37.837019 32411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:37.837059 32409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:37.836995 32408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:37.837921 32403 server_base.cc:1034] running on GCE node
I20260925 01:29:37.838794 32403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:37.840467 32403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:37.841707 32403 hybrid_clock.cc:648] HybridClock initialized: now 1790299777841664 us; error 39 us; skew 500 ppm
I20260925 01:29:37.842123 32403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:37 dist-test-slave-h6hf krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299777, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:37.846722 32403 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:37.851665 32403 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:37.851814 32403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:37.852496 32403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:37.858481 32403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "c134f4da23d941d9b5fce32b6aa75fc8"
format_stamp: "Formatted at 2026-09-25 01:29:37 on dist-test-slave-h6hf"
server_key: "394d6b80a9d94fda5149ecc8db72ecb6"
server_key_iv: "0d08981a0c858127433d1d596a2034a5"
server_key_version: "encryptionkey@0"
I20260925 01:29:37.859867 32403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "c134f4da23d941d9b5fce32b6aa75fc8"
format_stamp: "Formatted at 2026-09-25 01:29:37 on dist-test-slave-h6hf"
server_key: "394d6b80a9d94fda5149ecc8db72ecb6"
server_key_iv: "0d08981a0c858127433d1d596a2034a5"
server_key_version: "encryptionkey@0"
I20260925 01:29:37.868052 32403 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260925 01:29:37.873174 32417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:37.874951 32403 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:29:37.875139 32403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "c134f4da23d941d9b5fce32b6aa75fc8"
format_stamp: "Formatted at 2026-09-25 01:29:37 on dist-test-slave-h6hf"
server_key: "394d6b80a9d94fda5149ecc8db72ecb6"
server_key_iv: "0d08981a0c858127433d1d596a2034a5"
server_key_version: "encryptionkey@0"
I20260925 01:29:37.875370 32403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:37.894868 32403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:37.899573 32403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:37.899878 32403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:37.918880 32403 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:43969
I20260925 01:29:37.918903 32468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:43969 every 8 connection(s)
I20260925 01:29:37.919824 32403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:37.923563 32469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:37.928004 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 32403
I20260925 01:29:37.928259 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:37.928876 28288 external_mini_cluster.cc:1428] Setting key 136741aa83f365f07b63c6e2f158c69c
I20260925 01:29:37.938549 32469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8: Bootstrap starting.
I20260925 01:29:37.945282 32469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:37.946866 32469 log.cc:824] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:37.951298 32469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8: No bootstrap required, opened a new log
Sep 25 01:29:37 dist-test-slave-h6hf krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:37.962281 32469 raft_consensus.cc:348] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c134f4da23d941d9b5fce32b6aa75fc8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43969 } }
I20260925 01:29:37.962796 32469 raft_consensus.cc:374] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:37.962924 32469 raft_consensus.cc:729] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c134f4da23d941d9b5fce32b6aa75fc8, State: Initialized, Role: FOLLOWER
I20260925 01:29:37.963649 32469 consensus_queue.cc:260] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c134f4da23d941d9b5fce32b6aa75fc8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43969 } }
I20260925 01:29:37.963933 32469 raft_consensus.cc:388] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:37.964092 32469 raft_consensus.cc:482] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:37.964289 32469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:37.966630 32472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:37.938986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.27.160.62:43969)
0925 01:29:37.939938 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:37.939958 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:37.940018 (+ 60us) server_negotiation.cc:407] Connection header received
0925 01:29:37.943355 (+ 3337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:37.943398 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:37.944048 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:37.947102 (+ 3054us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:37.947794 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:37.950088 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:37.952245 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:37.952906 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:37.953125 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:37.957626 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:37.957637 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:37.957659 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:37.957711 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:37.961749 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:37.962496 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:37.962508 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:37.962516 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:37.962736 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:37.963211 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:37.963217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:37.963220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:37.963983 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:37.964312 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:37.964998 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 01:29:37.965535 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":413,"threads_started":1}
I20260925 01:29:37.967172 32469 raft_consensus.cc:504] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c134f4da23d941d9b5fce32b6aa75fc8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43969 } }
I20260925 01:29:37.967696 32469 leader_election.cc:302] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c134f4da23d941d9b5fce32b6aa75fc8; no voters:
I20260925 01:29:37.968408 32469 leader_election.cc:288] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:37.968520 32474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:37.970445 32474 raft_consensus.cc:686] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [term 1 LEADER]: Becoming Leader. State: Replica: c134f4da23d941d9b5fce32b6aa75fc8, State: Running, Role: LEADER
I20260925 01:29:37.970644 32469 sys_catalog.cc:564] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:37.970973 32474 consensus_queue.cc:237] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c134f4da23d941d9b5fce32b6aa75fc8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43969 } }
I20260925 01:29:37.973687 32476 sys_catalog.cc:455] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c134f4da23d941d9b5fce32b6aa75fc8. Latest consensus state: current_term: 1 leader_uuid: "c134f4da23d941d9b5fce32b6aa75fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c134f4da23d941d9b5fce32b6aa75fc8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43969 } } }
I20260925 01:29:37.974193 32476 sys_catalog.cc:458] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:37.974040 32475 sys_catalog.cc:455] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c134f4da23d941d9b5fce32b6aa75fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c134f4da23d941d9b5fce32b6aa75fc8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43969 } } }
I20260925 01:29:37.974305 32475 sys_catalog.cc:458] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:37.977026 32484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:37.981609 32484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:37.989892 32484 catalog_manager.cc:1347] Generated new cluster ID: 5fb6cfa39d7841b68daf3175f2b43376
I20260925 01:29:37.990012 32484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:38.019644 32484 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:38.021725 32484 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:38.038748 32484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8: Generated new TSK 0
I20260925 01:29:38.039858 32484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:29:38.101670 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43969
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:38.430859 32495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:38.431164 32495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:38.431262 32495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:38.438840 32495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:38.438966 32495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:38.439087 32495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:38.450111 32495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:38.451400 32495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:38.452704 32495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:38.466066 32501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:38.466023 32500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:38.466290 32503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:38.467000 32495 server_base.cc:1034] running on GCE node
I20260925 01:29:38.468102 32495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:38.469267 32495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:38.470522 32495 hybrid_clock.cc:648] HybridClock initialized: now 1790299778470464 us; error 65 us; skew 500 ppm
I20260925 01:29:38.471009 32495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:38 dist-test-slave-h6hf krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299778, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:38.474802 32495 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:29:38.476207 32495 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:38.476342 32495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:38.476708 32495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:38.479708 32495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "103a563370124edbb829d773f59ee863"
format_stamp: "Formatted at 2026-09-25 01:29:38 on dist-test-slave-h6hf"
server_key: "a7326910d7687bbcd30320247777139e"
server_key_iv: "c8f0ae323b9ec0958ede4dc70dc78a14"
server_key_version: "encryptionkey@0"
I20260925 01:29:38.480535 32495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "103a563370124edbb829d773f59ee863"
format_stamp: "Formatted at 2026-09-25 01:29:38 on dist-test-slave-h6hf"
server_key: "a7326910d7687bbcd30320247777139e"
server_key_iv: "c8f0ae323b9ec0958ede4dc70dc78a14"
server_key_version: "encryptionkey@0"
I20260925 01:29:38.485810 32495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:29:38.489245 32509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:38.490617 32495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:38.490809 32495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "103a563370124edbb829d773f59ee863"
format_stamp: "Formatted at 2026-09-25 01:29:38 on dist-test-slave-h6hf"
server_key: "a7326910d7687bbcd30320247777139e"
server_key_iv: "c8f0ae323b9ec0958ede4dc70dc78a14"
server_key_version: "encryptionkey@0"
I20260925 01:29:38.491056 32495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:38.503278 32495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:38.507303 32495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:38.507579 32495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:38.508558 32495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:38.509971 32495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:38.510092 32495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:38.510219 32495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:38.510298 32495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:38.544245 32495 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:39953
I20260925 01:29:38.544302 32621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:39953 every 8 connection(s)
I20260925 01:29:38.545171 32495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:38.551841 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 32495
I20260925 01:29:38.552067 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:38.552757 28288 external_mini_cluster.cc:1428] Setting key 8d18433afd425196f9290a0e5d5d39b4
Sep 25 01:29:38 dist-test-slave-h6hf krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299778, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:38.565992 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:38.547668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:54861 (local address 127.27.160.62:43969)
0925 01:29:38.548118 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:38.548124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:38.549205 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 01:29:38.550988 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:38.550992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:38.551058 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:38.551159 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:38.552749 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:38.553789 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:38.555717 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:38.556514 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:38.556676 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:38.560118 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:38.560124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:38.560128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:38.560161 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:38.562842 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:38.563602 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:38.563610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:38.563614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:38.563741 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:38.564175 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:38.564181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:38.564184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:38.564454 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:38.564622 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:38.565388 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 01:29:38.565631 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260925 01:29:38.567047 32625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:38.548300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43969 (local address 127.27.160.1:54861)
0925 01:29:38.548991 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:38.549053 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:38.550732 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:38.551365 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:38.551374 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:38.551873 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:38.552568 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:38.552587 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:38.553964 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:38.553970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:38.555560 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:38.555572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:38.556733 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:38.556738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:38.556919 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:38.557666 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:38.557693 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:38.559874 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:38.562997 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:38.563007 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:38.563026 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:38.563469 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:38.563857 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:38.563862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:38.563865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:38.564060 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:38.564590 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:38.564599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:38.564958 (+ 359us) client_negotiation.cc:769] Sending connection context
0925 01:29:38.565299 (+ 341us) client_negotiation.cc:240] Negotiation successful
0925 01:29:38.565844 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":276,"threads_started":1}
I20260925 01:29:38.568863 32622 heartbeater.cc:347] Connected to a master server at 127.27.160.62:43969
I20260925 01:29:38.569243 32622 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:38.569924 32622 heartbeater.cc:511] Master 127.27.160.62:43969 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:29:38.572578 32434 ts_manager.cc:194] Registered new tserver with Master: 103a563370124edbb829d773f59ee863 (127.27.160.1:39953)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
I20260925 01:29:38.574484 32434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:54861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:38.598532 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43969
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:38.922113 32628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:38.922487 32628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:38.922658 32628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:38.934217 32628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:38.934355 32628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:38.934543 32628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:38.946681 32628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:38.947926 32628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:38.949223 32628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:38.961800 32633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:38.961871 32634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:38.961848 32636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:38.962646 32628 server_base.cc:1034] running on GCE node
I20260925 01:29:38.963264 32628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:38.964200 32628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:38.965399 32628 hybrid_clock.cc:648] HybridClock initialized: now 1790299778965349 us; error 45 us; skew 500 ppm
I20260925 01:29:38.965745 32628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:38 dist-test-slave-h6hf krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299778, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:38.969086 32628 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:38.970304 32628 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:38.970393 32628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:38.970702 32628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:38.973964 32628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "1d2b02010e3c499a9a83a8eea6738a65"
format_stamp: "Formatted at 2026-09-25 01:29:38 on dist-test-slave-h6hf"
server_key: "4c729595e0360314a2f6cb8289928da9"
server_key_iv: "9bac45333a74294563a8d8b007bf6fa4"
server_key_version: "encryptionkey@0"
I20260925 01:29:38.974786 32628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "1d2b02010e3c499a9a83a8eea6738a65"
format_stamp: "Formatted at 2026-09-25 01:29:38 on dist-test-slave-h6hf"
server_key: "4c729595e0360314a2f6cb8289928da9"
server_key_iv: "9bac45333a74294563a8d8b007bf6fa4"
server_key_version: "encryptionkey@0"
I20260925 01:29:38.979982 32628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:29:38.983968 32642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:38.985445 32628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:38.985632 32628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "1d2b02010e3c499a9a83a8eea6738a65"
format_stamp: "Formatted at 2026-09-25 01:29:38 on dist-test-slave-h6hf"
server_key: "4c729595e0360314a2f6cb8289928da9"
server_key_iv: "9bac45333a74294563a8d8b007bf6fa4"
server_key_version: "encryptionkey@0"
I20260925 01:29:38.985824 32628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:39.007436 32628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:39.012137 32628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:39.012434 32628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:39.013566 32628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:39.015092 32628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:39.015185 32628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:39.015303 32628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:39.015373 32628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:39.050495 32628 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:33995
I20260925 01:29:39.050589 32754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:33995 every 8 connection(s)
I20260925 01:29:39.051405 32628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:39.058748 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 32628
I20260925 01:29:39.059010 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:39.059582 28288 external_mini_cluster.cc:1428] Setting key 6658bfbfca1c293e88dce1a8a3b8a783
Sep 25 01:29:39 dist-test-slave-h6hf krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299778, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:39.074928 32624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.054128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:53213 (local address 127.27.160.62:43969)
0925 01:29:39.054311 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.054316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.055522 (+ 1206us) server_negotiation.cc:407] Connection header received
0925 01:29:39.057604 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.057611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.057715 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.057847 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:39.060072 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.060858 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.063576 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.064313 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.064438 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.068653 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:39.068658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:39.068662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:39.068693 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:39.071619 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:39.072531 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:39.072541 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:39.072543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:39.072670 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:39.073108 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:39.073161 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:39.073165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:39.073479 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:39.073653 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.074354 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.074583 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:29:39.075959 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.054438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43969 (local address 127.27.160.2:53213)
0925 01:29:39.055290 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:39.055350 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:39.057301 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:39.058107 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:39.058116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:39.058943 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:39.059866 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.059894 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.061040 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.061046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.063355 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.063386 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.064725 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.064734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.065025 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.065881 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:39.065917 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:39.068398 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:39.071765 (+ 3367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:39.071773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:39.071791 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:39.072394 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:39.072783 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:39.072789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:39.072793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:39.072982 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:39.073628 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:39.073636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:39.073989 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 01:29:39.074255 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 01:29:39.074831 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":291,"threads_started":1}
I20260925 01:29:39.077514 32755 heartbeater.cc:347] Connected to a master server at 127.27.160.62:43969
I20260925 01:29:39.077924 32755 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:39.078836 32755 heartbeater.cc:511] Master 127.27.160.62:43969 requested a full tablet report, sending...
I20260925 01:29:39.080577 32434 ts_manager.cc:194] Registered new tserver with Master: 1d2b02010e3c499a9a83a8eea6738a65 (127.27.160.2:33995)
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:29:39.081954 32434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:53213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:39.108893 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43969
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:39.434841 32760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:39.435127 32760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:39.435227 32760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:39.443264 32760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:39.443394 32760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:39.443512 32760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:39.453641 32760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:39.454924 32760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:39.456210 32760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:39.469280 32765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:39.469350 32766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:39.469208 300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:39.470713 32760 server_base.cc:1034] running on GCE node
I20260925 01:29:39.471364 32760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:39.472481 32760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:39.473721 32760 hybrid_clock.cc:648] HybridClock initialized: now 1790299779473649 us; error 72 us; skew 500 ppm
I20260925 01:29:39.474073 32760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:39 dist-test-slave-h6hf krb5kdc[32390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299779, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:39.477382 32760 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:39.478565 32760 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:39.478703 32760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:39.479020 32760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:39.482097 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "1a231339064f428f90987067c7f8387b"
format_stamp: "Formatted at 2026-09-25 01:29:39 on dist-test-slave-h6hf"
server_key: "8d30acee4f224a4c8d4e2369b5fb5f36"
server_key_iv: "b6d97445759f092dfdf1efb694b1430b"
server_key_version: "encryptionkey@0"
I20260925 01:29:39.482952 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "1a231339064f428f90987067c7f8387b"
format_stamp: "Formatted at 2026-09-25 01:29:39 on dist-test-slave-h6hf"
server_key: "8d30acee4f224a4c8d4e2369b5fb5f36"
server_key_iv: "b6d97445759f092dfdf1efb694b1430b"
server_key_version: "encryptionkey@0"
I20260925 01:29:39.488400 32760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:29:39.492192 306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:39.493695 32760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:39.493872 32760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "1a231339064f428f90987067c7f8387b"
format_stamp: "Formatted at 2026-09-25 01:29:39 on dist-test-slave-h6hf"
server_key: "8d30acee4f224a4c8d4e2369b5fb5f36"
server_key_iv: "b6d97445759f092dfdf1efb694b1430b"
server_key_version: "encryptionkey@0"
I20260925 01:29:39.494110 32760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:39.526875 32760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:39.532343 32760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:39.532672 32760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:39.533887 32760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:39.535691 32760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:39.535812 32760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:39.535984 32760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:39.536078 32760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:39.571938 32760 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:36189
I20260925 01:29:39.572049 418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:36189 every 8 connection(s)
I20260925 01:29:39.572968 32760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:39.577793 32622 heartbeater.cc:503] Master 127.27.160.62:43969 was elected leader, sending a full tablet report...
I20260925 01:29:39.579471 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 32760
I20260925 01:29:39.579710 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:39.580150 28288 external_mini_cluster.cc:1428] Setting key a71a86c465086066a76409439fd1751c
Sep 25 01:29:39 dist-test-slave-h6hf krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299779, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:39.593540 421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.575725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:53839 (local address 127.27.160.62:43969)
0925 01:29:39.576179 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.576185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.577277 (+ 1092us) server_negotiation.cc:407] Connection header received
0925 01:29:39.579411 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.579417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.579500 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.579709 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:39.581512 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.582392 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.584223 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.584657 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.584807 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.588327 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:39.588333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:39.588337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:39.588369 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:39.590834 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:39.591531 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:39.591534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:39.591536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:39.591650 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:39.592028 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:39.592031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:39.592033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:39.592275 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:39.592385 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.593046 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.593270 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260925 01:29:39.594884 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.576327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43969 (local address 127.27.160.3:53839)
0925 01:29:39.577058 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:39.577104 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:39.579106 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:39.579959 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:39.579972 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:39.580587 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:39.581316 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.581344 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.582554 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.582561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.584046 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.584058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.584912 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.584919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.585217 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.585870 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:39.585894 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:39.588094 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:39.590972 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:39.590979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:39.590993 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:39.591422 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:39.591740 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:39.591743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:39.591745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:39.591919 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:39.592379 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:39.592385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:39.592714 (+ 329us) client_negotiation.cc:769] Sending connection context
0925 01:29:39.592950 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 01:29:39.593409 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":287,"threads_started":1}
I20260925 01:29:39.596431 419 heartbeater.cc:347] Connected to a master server at 127.27.160.62:43969
I20260925 01:29:39.596750 419 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:39.597489 419 heartbeater.cc:511] Master 127.27.160.62:43969 requested a full tablet report, sending...
I20260925 01:29:39.598973 32433 ts_manager.cc:194] Registered new tserver with Master: 1a231339064f428f90987067c7f8387b (127.27.160.3:36189)
I20260925 01:29:39.599872 32433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:53839
I20260925 01:29:39.608738 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:29:39.625695 421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.613525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.27.160.62:43969)
0925 01:29:39.613707 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.613711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.614136 (+ 425us) server_negotiation.cc:407] Connection header received
0925 01:29:39.614273 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.614277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.614339 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.614423 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:39.615794 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.616498 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.617819 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.618244 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.618348 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.620647 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:39.620652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:39.620655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:39.620690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:39.623321 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:39.623998 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:39.624002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:39.624004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:39.624128 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:39.624448 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:39.624451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:39.624453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:39.624787 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:39.625000 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.625254 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.625441 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:29:39.630133 32433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:39.636926 32433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:39.664940 432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.648970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:46744)
0925 01:29:39.651827 (+ 2857us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:39.651851 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:39.652010 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:39.653280 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:39.653293 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:39.653336 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:39.653925 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.653949 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.656577 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.656586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.659963 (+ 3377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.659991 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.663765 (+ 3774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.663772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.664006 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.664103 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 01:29:39.664266 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 01:29:39.664493 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2685,"thread_start_us":1195,"threads_started":1}
I20260925 01:29:39.665685 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.649517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.27.160.1:39953)
0925 01:29:39.651109 (+ 1592us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.651122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.652166 (+ 1044us) server_negotiation.cc:407] Connection header received
0925 01:29:39.652324 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.652335 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.652687 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.652924 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:39.654198 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.656291 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.660258 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.663524 (+ 3266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.663717 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.663980 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.665005 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.665343 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":2663,"threads_started":1}
I20260925 01:29:39.669984 32557 tablet_service.cc:1467] Processing CreateTablet for tablet 618bb44ec31a45bb9dff5a1b4f2f424e (DEFAULT_TABLE table=test-table [id=ceb9889aeba04198b161639ea58101f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:39.671372 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.648929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:36189 (local address 127.0.0.1:37312)
0925 01:29:39.650018 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:39.650079 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:39.650307 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:39.664438 (+ 14131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:39.664451 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:39.664481 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:39.664872 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.664888 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.667046 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.667052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.669074 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.669086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.670524 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.670531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.670697 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.670735 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 01:29:39.670832 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 01:29:39.670985 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":2479,"threads_started":1}
I20260925 01:29:39.671801 32557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618bb44ec31a45bb9dff5a1b4f2f424e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:39.672657 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.649207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.27.160.3:36189)
0925 01:29:39.663633 (+ 14426us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.663645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.663692 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:29:39.663823 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.663834 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.664185 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.665091 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:39.665431 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.666888 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.669275 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.670272 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.670462 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.670697 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.670931 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.671125 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14184,"thread_start_us":13970,"threads_started":1}
I20260925 01:29:39.673009 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.651516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:33995 (local address 127.0.0.1:56182)
0925 01:29:39.664463 (+ 12947us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:39.664483 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:39.664617 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:39.665768 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:39.665772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:39.665799 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:39.666223 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.666236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.668109 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.668114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.670225 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.670240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.672273 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.672280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.672441 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.672475 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 01:29:39.672575 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 01:29:39.672714 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12801,"thread_start_us":12655,"threads_started":1}
I20260925 01:29:39.673558 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.652029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.27.160.2:33995)
0925 01:29:39.662962 (+ 10933us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.662978 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.664771 (+ 1793us) server_negotiation.cc:407] Connection header received
0925 01:29:39.664878 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.664885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.665586 (+ 701us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.665778 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:39.666407 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.667939 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.670388 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.672054 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.672195 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.672370 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.672941 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.673108 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9939,"thread_start_us":10660,"threads_started":1}
I20260925 01:29:39.676702 32690 tablet_service.cc:1467] Processing CreateTablet for tablet 618bb44ec31a45bb9dff5a1b4f2f424e (DEFAULT_TABLE table=test-table [id=ceb9889aeba04198b161639ea58101f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:39.676790 354 tablet_service.cc:1467] Processing CreateTablet for tablet 618bb44ec31a45bb9dff5a1b4f2f424e (DEFAULT_TABLE table=test-table [id=ceb9889aeba04198b161639ea58101f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:39.678107 32690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618bb44ec31a45bb9dff5a1b4f2f424e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:39.678203 354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618bb44ec31a45bb9dff5a1b4f2f424e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:39.688328 438 tablet_bootstrap.cc:492] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Bootstrap starting.
I20260925 01:29:39.689697 439 tablet_bootstrap.cc:492] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Bootstrap starting.
I20260925 01:29:39.692204 438 tablet_bootstrap.cc:654] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:39.692689 440 tablet_bootstrap.cc:492] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: Bootstrap starting.
I20260925 01:29:39.693225 439 tablet_bootstrap.cc:654] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:39.693856 438 log.cc:824] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:39.694365 439 log.cc:824] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:39.696719 440 tablet_bootstrap.cc:654] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:39.698127 440 log.cc:824] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:39.700100 439 tablet_bootstrap.cc:492] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: No bootstrap required, opened a new log
I20260925 01:29:39.700390 439 ts_tablet_manager.cc:1397] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 01:29:39.700737 438 tablet_bootstrap.cc:492] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: No bootstrap required, opened a new log
I20260925 01:29:39.701023 438 ts_tablet_manager.cc:1397] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260925 01:29:39.701180 440 tablet_bootstrap.cc:492] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: No bootstrap required, opened a new log
I20260925 01:29:39.701447 440 ts_tablet_manager.cc:1397] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 01:29:39.708057 439 raft_consensus.cc:348] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.708447 439 raft_consensus.cc:374] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:39.708572 439 raft_consensus.cc:729] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2b02010e3c499a9a83a8eea6738a65, State: Initialized, Role: FOLLOWER
I20260925 01:29:39.709208 439 consensus_queue.cc:260] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.710402 32755 heartbeater.cc:503] Master 127.27.160.62:43969 was elected leader, sending a full tablet report...
I20260925 01:29:39.711087 439 ts_tablet_manager.cc:1428] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260925 01:29:39.712328 438 raft_consensus.cc:348] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.712328 440 raft_consensus.cc:348] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.712781 440 raft_consensus.cc:374] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:39.712781 438 raft_consensus.cc:374] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:39.712894 438 raft_consensus.cc:729] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103a563370124edbb829d773f59ee863, State: Initialized, Role: FOLLOWER
I20260925 01:29:39.712889 440 raft_consensus.cc:729] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a231339064f428f90987067c7f8387b, State: Initialized, Role: FOLLOWER
I20260925 01:29:39.713549 438 consensus_queue.cc:260] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.713550 440 consensus_queue.cc:260] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.716625 419 heartbeater.cc:503] Master 127.27.160.62:43969 was elected leader, sending a full tablet report...
I20260925 01:29:39.718024 440 ts_tablet_manager.cc:1428] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260925 01:29:39.719157 438 ts_tablet_manager.cc:1428] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260925 01:29:39.754513 446 raft_consensus.cc:482] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:39.754891 446 raft_consensus.cc:504] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.756558 446 leader_election.cc:288] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 103a563370124edbb829d773f59ee863 (127.27.160.1:39953), 1d2b02010e3c499a9a83a8eea6738a65 (127.27.160.2:33995)
I20260925 01:29:39.763226 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.757110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:60227 (local address 127.27.160.1:39953)
0925 01:29:39.757668 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.757677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.757755 (+ 78us) server_negotiation.cc:407] Connection header received
0925 01:29:39.757941 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.757948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.758025 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.758120 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:39.758681 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.759420 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.761367 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.762096 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.762217 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.762271 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.762835 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.762955 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260925 01:29:39.763362 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.757015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:48235 (local address 127.27.160.2:33995)
0925 01:29:39.757759 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.757763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.757778 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:29:39.757834 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.757838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.757890 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.757967 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:39.758415 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.759252 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.761112 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.762243 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.762444 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.762534 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.762826 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.763041 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589}
I20260925 01:29:39.763239 422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.756878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:33995 (local address 127.27.160.3:48235)
0925 01:29:39.757281 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:39.757306 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:39.757457 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:39.757999 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:39.758003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:39.758035 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:39.758253 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.758273 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.759408 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.759412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.760842 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.760853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.762415 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.762419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.762631 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.762677 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 01:29:39.762751 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 01:29:39.762904 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":72}
I20260925 01:29:39.763784 447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.756948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.27.160.3:60227)
0925 01:29:39.757635 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:39.757652 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:39.757795 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:39.758141 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:39.758145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:39.758164 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:39.758530 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.758541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.759567 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.759571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.761214 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:39.761232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.762329 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.762340 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:39.762625 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.762670 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 01:29:39.762805 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 01:29:39.762957 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":8,"thread_start_us":395,"threads_started":1}
I20260925 01:29:39.764698 32710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618bb44ec31a45bb9dff5a1b4f2f424e" candidate_uuid: "1a231339064f428f90987067c7f8387b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2b02010e3c499a9a83a8eea6738a65" is_pre_election: true
I20260925 01:29:39.765002 32577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618bb44ec31a45bb9dff5a1b4f2f424e" candidate_uuid: "1a231339064f428f90987067c7f8387b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103a563370124edbb829d773f59ee863" is_pre_election: true
I20260925 01:29:39.765281 32710 raft_consensus.cc:2454] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a231339064f428f90987067c7f8387b in term 0.
I20260925 01:29:39.765681 32577 raft_consensus.cc:2454] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a231339064f428f90987067c7f8387b in term 0.
I20260925 01:29:39.765942 310 leader_election.cc:302] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a231339064f428f90987067c7f8387b, 1d2b02010e3c499a9a83a8eea6738a65; no voters:
I20260925 01:29:39.766377 446 raft_consensus.cc:2781] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:39.766559 446 raft_consensus.cc:482] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:39.766696 446 raft_consensus.cc:3037] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:39.769366 446 raft_consensus.cc:504] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.770150 446 leader_election.cc:288] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [CANDIDATE]: Term 1 election: Requested vote from peers 103a563370124edbb829d773f59ee863 (127.27.160.1:39953), 1d2b02010e3c499a9a83a8eea6738a65 (127.27.160.2:33995)
I20260925 01:29:39.770614 32577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618bb44ec31a45bb9dff5a1b4f2f424e" candidate_uuid: "1a231339064f428f90987067c7f8387b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "103a563370124edbb829d773f59ee863"
I20260925 01:29:39.770768 32710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618bb44ec31a45bb9dff5a1b4f2f424e" candidate_uuid: "1a231339064f428f90987067c7f8387b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2b02010e3c499a9a83a8eea6738a65"
I20260925 01:29:39.770912 32577 raft_consensus.cc:3037] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:39.771013 32710 raft_consensus.cc:3037] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:39.774747 32710 raft_consensus.cc:2454] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a231339064f428f90987067c7f8387b in term 1.
I20260925 01:29:39.774747 32577 raft_consensus.cc:2454] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a231339064f428f90987067c7f8387b in term 1.
I20260925 01:29:39.775231 307 leader_election.cc:302] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 103a563370124edbb829d773f59ee863, 1a231339064f428f90987067c7f8387b; no voters:
I20260925 01:29:39.775557 446 raft_consensus.cc:2781] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:39.775986 446 raft_consensus.cc:686] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 1 LEADER]: Becoming Leader. State: Replica: 1a231339064f428f90987067c7f8387b, State: Running, Role: LEADER
I20260925 01:29:39.776479 446 consensus_queue.cc:237] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } }
I20260925 01:29:39.781570 32433 catalog_manager.cc:5495] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a231339064f428f90987067c7f8387b (127.27.160.3). New cstate: current_term: 1 leader_uuid: "1a231339064f428f90987067c7f8387b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a231339064f428f90987067c7f8387b" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 } health_report { overall_health: UNKNOWN } } }
W20260925 01:29:39.797434 32623 tablet.cc:2369] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:39.801052 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.795909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.27.160.3:36189)
0925 01:29:39.796091 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.796096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.796113 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:29:39.796264 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.796267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.796342 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.796480 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:39.796798 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.797801 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.799233 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.799627 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.799720 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.800113 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:39.800601 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:39.800710 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.800780 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.800843 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 01:29:39.803756 32756 tablet.cc:2369] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:39.809495 32710 raft_consensus.cc:1261] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 1 FOLLOWER]: Refusing update from remote peer 1a231339064f428f90987067c7f8387b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:39.809508 32577 raft_consensus.cc:1261] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 1 FOLLOWER]: Refusing update from remote peer 1a231339064f428f90987067c7f8387b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:39.810520 446 consensus_queue.cc:1035] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [LEADER]: Connected to new peer: Peer: permanent_uuid: "103a563370124edbb829d773f59ee863" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 39953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:39.810860 448 consensus_queue.cc:1035] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d2b02010e3c499a9a83a8eea6738a65" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 33995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:39.822798 451 mvcc.cc:205] Tried to move back new op lower bound from 7333067898089758720 to 7333067897968578560. Current Snapshot: MvccSnapshot[applied={T|T < 7333067898089758720}]
I20260925 01:29:39.836583 32433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:39.836867 32433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:39.840629 32433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c134f4da23d941d9b5fce32b6aa75fc8: Sending DeleteTablet for 3 replicas of tablet 618bb44ec31a45bb9dff5a1b4f2f424e
I20260925 01:29:39.841794 354 tablet_service.cc:1514] Processing DeleteTablet for tablet 618bb44ec31a45bb9dff5a1b4f2f424e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:39 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:37312
I20260925 01:29:39.841915 32690 tablet_service.cc:1514] Processing DeleteTablet for tablet 618bb44ec31a45bb9dff5a1b4f2f424e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:39 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:56182
I20260925 01:29:39.842624 32557 tablet_service.cc:1514] Processing DeleteTablet for tablet 618bb44ec31a45bb9dff5a1b4f2f424e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:39 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:46744
I20260925 01:29:39.843724 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 32495
I20260925 01:29:39.850234 463 tablet_replica.cc:330] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: stopping tablet replica
I20260925 01:29:39.850871 463 raft_consensus.cc:2229] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:39.851547 463 raft_consensus.cc:2258] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:39.852612 462 tablet_replica.cc:330] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: stopping tablet replica
I20260925 01:29:39.853212 462 raft_consensus.cc:2229] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:39.853861 462 raft_consensus.cc:2258] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:39.856395 463 ts_tablet_manager.cc:1905] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 01:29:39 dist-test-slave-h6hf krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:39.864558 462 ts_tablet_manager.cc:1905] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:39.864668 461 tablet_replica.cc:330] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: stopping tablet replica
I20260925 01:29:39.865151 461 raft_consensus.cc:2229] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:39.865865 461 raft_consensus.cc:2258] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:39.868091 461 ts_tablet_manager.cc:1905] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:39.870601 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:39.845530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.27.160.1:39953)
0925 01:29:39.845763 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:39.845772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:39.849680 (+ 3908us) server_negotiation.cc:407] Connection header received
0925 01:29:39.850018 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:39.850027 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:39.850138 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:39.850236 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:39.852465 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.853666 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.856163 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:39.856817 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:39.856923 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:39.862332 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:39.862349 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:39.862368 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:39.862415 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:39.866757 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:39.867798 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:39.867810 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:39.867814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:39.867987 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:39.868477 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:39.868483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:39.868486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:39.868965 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:39.869297 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:39.869949 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 01:29:39.870183 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:29:39.872228 32617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:39.874400 462 ts_tablet_manager.cc:1918] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:39.874621 462 log.cc:1196] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/618bb44ec31a45bb9dff5a1b4f2f424e
I20260925 01:29:39.875200 462 ts_tablet_manager.cc:1939] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1d2b02010e3c499a9a83a8eea6738a65: Deleting consensus metadata
I20260925 01:29:39.876076 461 ts_tablet_manager.cc:1918] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:39.876252 461 log.cc:1196] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/618bb44ec31a45bb9dff5a1b4f2f424e
I20260925 01:29:39.876827 461 ts_tablet_manager.cc:1939] T 618bb44ec31a45bb9dff5a1b4f2f424e P 1a231339064f428f90987067c7f8387b: Deleting consensus metadata
I20260925 01:29:39.877420 32421 catalog_manager.cc:4841] TS 1d2b02010e3c499a9a83a8eea6738a65 (127.27.160.2:33995): tablet 618bb44ec31a45bb9dff5a1b4f2f424e (table test-table [id=ceb9889aeba04198b161639ea58101f0]) successfully deleted
I20260925 01:29:39.878840 32419 catalog_manager.cc:4841] TS 1a231339064f428f90987067c7f8387b (127.27.160.3:36189): tablet 618bb44ec31a45bb9dff5a1b4f2f424e (table test-table [id=ceb9889aeba04198b161639ea58101f0]) successfully deleted
I20260925 01:29:39.983640 463 ts_tablet_manager.cc:1918] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:39.983856 463 log.cc:1196] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/618bb44ec31a45bb9dff5a1b4f2f424e
I20260925 01:29:39.984393 463 ts_tablet_manager.cc:1939] T 618bb44ec31a45bb9dff5a1b4f2f424e P 103a563370124edbb829d773f59ee863: Deleting consensus metadata
I20260925 01:29:39.985877 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 32495
I20260925 01:29:40.004101 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 32628
W20260925 01:29:40.004101 32418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.160.1:39953 (error 108)
I20260925 01:29:40.006116 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.005618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:46754)
0925 01:29:40.005810 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:40.005939 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 01:29:40.006518 32418 catalog_manager.cc:4584] TS 103a563370124edbb829d773f59ee863 (127.27.160.1:39953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 618bb44ec31a45bb9dff5a1b4f2f424e: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Sep 25 01:29:40 dist-test-slave-h6hf krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:29:40.020401 437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.005789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.27.160.2:33995)
0925 01:29:40.005969 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:40.005973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:40.005990 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:29:40.006045 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:40.006048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:40.006117 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:40.006235 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:40.007564 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:40.008303 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:40.009773 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:40.010161 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:40.010301 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:40.014619 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:40.014634 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:40.014649 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:40.014684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:40.017601 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:40.018298 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:40.018306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:40.018308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:40.018432 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:40.018838 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:40.018843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:40.018845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:40.019219 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:40.019525 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:40.019831 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 01:29:40.020066 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:29:40.021209 32750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:40.060287 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.059712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:46758)
0925 01:29:40.059965 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:40.060096 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 01:29:40.061502 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.061075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:46766)
0925 01:29:40.061253 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:40.061341 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 01:29:40.113186 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 32628
I20260925 01:29:40.122115 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.121632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:46782)
0925 01:29:40.121860 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:40.121948 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 01:29:40.123147 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.122818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:46794)
0925 01:29:40.122963 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:40.123038 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 01:29:40.135946 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 32760
Sep 25 01:29:40 dist-test-slave-h6hf krb5kdc[32390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:29:40.153873 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.136729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.27.160.3:36189)
0925 01:29:40.136925 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:40.136942 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:40.136969 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:29:40.137049 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:40.137054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:40.137170 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:40.137292 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:40.138476 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:40.139680 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:40.141045 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:40.141927 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:40.142071 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:40.146478 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:40.146495 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:40.146517 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:40.146570 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:40.150924 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:40.151587 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:40.151596 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:40.151600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:40.151778 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:40.152179 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:40.152185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:40.152189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:40.152666 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:40.152979 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:40.153237 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 01:29:40.153462 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:29:40.154587 414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:40.203980 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.203482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:46802)
0925 01:29:40.203710 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:40.203806 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 01:29:40.205078 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.204684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:46806)
0925 01:29:40.204873 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:40.204968 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 01:29:40.283181 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 32760
I20260925 01:29:40.302475 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 32403
I20260925 01:29:40.303664 32464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:40.413787 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.413154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:39953 (local address 127.0.0.1:35526)
0925 01:29:40.413375 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:40.413524 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:39953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 01:29:40.413964 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 32403
2026-09-25T01:29:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4331 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T01:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:40Z Disabled control of system clock
I20260925 01:29:40.481954 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40233
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 01:29:40.807925 473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:40.808226 473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:40.808311 473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:40.815579 473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:40.815697 473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:40.815737 473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:40.815795 473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:40.815838 473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:40.825001 473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:40233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:40.826329 473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:40.828090 473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:40.840251 478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:40.840337 479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:40.840293 481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:40.841168 473 server_base.cc:1034] running on GCE node
I20260925 01:29:40.841992 473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:40.843384 473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:40.844604 473 hybrid_clock.cc:648] HybridClock initialized: now 1790299780844544 us; error 57 us; skew 500 ppm
I20260925 01:29:40.845057 473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:40.850922 473 webserver.cc:460] Webserver started at http://127.27.160.62:45981/ using document root <none> and password file <none>
I20260925 01:29:40.851711 473 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:40.851835 473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:40.852196 473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:40.855449 473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "87796fc161e941f784f62e31f24db780"
format_stamp: "Formatted at 2026-09-25 01:29:40 on dist-test-slave-h6hf"
server_key: "4c51baa61937606ebe2a3a98ec5ae2a7"
server_key_iv: "7be49272bed0ae6329564f374804b613"
server_key_version: "encryptionkey@0"
I20260925 01:29:40.856345 473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "87796fc161e941f784f62e31f24db780"
format_stamp: "Formatted at 2026-09-25 01:29:40 on dist-test-slave-h6hf"
server_key: "4c51baa61937606ebe2a3a98ec5ae2a7"
server_key_iv: "7be49272bed0ae6329564f374804b613"
server_key_version: "encryptionkey@0"
I20260925 01:29:40.862250 473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:29:40.866225 487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:40.867719 473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260925 01:29:40.867940 473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "87796fc161e941f784f62e31f24db780"
format_stamp: "Formatted at 2026-09-25 01:29:40 on dist-test-slave-h6hf"
server_key: "4c51baa61937606ebe2a3a98ec5ae2a7"
server_key_iv: "7be49272bed0ae6329564f374804b613"
server_key_version: "encryptionkey@0"
I20260925 01:29:40.868178 473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:40.874050 473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:40.874817 473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:40.875180 473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:40.893499 473 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:40233
I20260925 01:29:40.893517 538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:40233 every 8 connection(s)
I20260925 01:29:40.894904 473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:40.898650 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:40.901224 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 473
I20260925 01:29:40.901419 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:40.902081 28288 external_mini_cluster.cc:1428] Setting key 667b908c331d4a44940010b2c670c88d
I20260925 01:29:40.910580 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:40.904145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.27.160.62:40233)
0925 01:29:40.905227 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:40.905238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:40.905279 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:29:40.906954 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:40.906977 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:40.907418 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:40.907833 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:40.908947 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:40.908954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:40.908966 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:40.908983 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:40.909038 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:40.909053 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:40.909303 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:40.909453 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 01:29:40.909708 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":332,"threads_started":1}
I20260925 01:29:40.910719 539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780: Bootstrap starting.
I20260925 01:29:40.914216 539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:40.915331 539 log.cc:824] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:40.918226 539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780: No bootstrap required, opened a new log
I20260925 01:29:40.925541 539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87796fc161e941f784f62e31f24db780" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40233 } }
I20260925 01:29:40.925920 539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:40.926033 539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87796fc161e941f784f62e31f24db780, State: Initialized, Role: FOLLOWER
I20260925 01:29:40.926617 539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87796fc161e941f784f62e31f24db780" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40233 } }
I20260925 01:29:40.926820 539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:40.926990 539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:40.927199 539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:40.929505 539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87796fc161e941f784f62e31f24db780" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40233 } }
I20260925 01:29:40.930019 539 leader_election.cc:302] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87796fc161e941f784f62e31f24db780; no voters:
I20260925 01:29:40.930647 539 leader_election.cc:288] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:40.930804 544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:40.931509 544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [term 1 LEADER]: Becoming Leader. State: Replica: 87796fc161e941f784f62e31f24db780, State: Running, Role: LEADER
I20260925 01:29:40.932073 544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87796fc161e941f784f62e31f24db780" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40233 } }
I20260925 01:29:40.932780 539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:40.934922 545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87796fc161e941f784f62e31f24db780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87796fc161e941f784f62e31f24db780" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40233 } } }
I20260925 01:29:40.935379 545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:40.935165 546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87796fc161e941f784f62e31f24db780. Latest consensus state: current_term: 1 leader_uuid: "87796fc161e941f784f62e31f24db780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87796fc161e941f784f62e31f24db780" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40233 } } }
I20260925 01:29:40.935570 546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:40.938272 552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:40.943543 552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:40.952347 552 catalog_manager.cc:1347] Generated new cluster ID: bab944cfb88247f982bb1243fe7aff00
I20260925 01:29:40.952468 552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:40.970649 552 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:40.971143 552 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:40.980634 552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780: Generated new TSK 0
I20260925 01:29:40.981711 552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:29:40.995127 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40233
--builtin_ntp_servers=127.27.160.20:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 01:29:41.325332 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:41.325628 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:41.325803 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:41.333318 563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:41.333461 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:41.333657 563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:41.343099 563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:41.344506 563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:41.346320 563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:41.359464 571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:41.359524 569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:41.359555 568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:41.365485 563 server_base.cc:1034] running on GCE node
I20260925 01:29:41.366146 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:41.367093 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:41.368300 563 hybrid_clock.cc:648] HybridClock initialized: now 1790299781368260 us; error 38 us; skew 500 ppm
I20260925 01:29:41.368690 563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:41.371208 563 webserver.cc:460] Webserver started at http://127.27.160.1:46545/ using document root <none> and password file <none>
I20260925 01:29:41.372035 563 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:41.372150 563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:41.372498 563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:41.375644 563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "5222a08177f04b1eb79754e9f387528e"
format_stamp: "Formatted at 2026-09-25 01:29:41 on dist-test-slave-h6hf"
server_key: "34fab711ddfa658a10f35bd2c330033b"
server_key_iv: "95cbcf1ac8c7e78d59f06006bde2bfdf"
server_key_version: "encryptionkey@0"
I20260925 01:29:41.376577 563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "5222a08177f04b1eb79754e9f387528e"
format_stamp: "Formatted at 2026-09-25 01:29:41 on dist-test-slave-h6hf"
server_key: "34fab711ddfa658a10f35bd2c330033b"
server_key_iv: "95cbcf1ac8c7e78d59f06006bde2bfdf"
server_key_version: "encryptionkey@0"
I20260925 01:29:41.381901 563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260925 01:29:41.385403 577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:41.386781 563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:29:41.386965 563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "5222a08177f04b1eb79754e9f387528e"
format_stamp: "Formatted at 2026-09-25 01:29:41 on dist-test-slave-h6hf"
server_key: "34fab711ddfa658a10f35bd2c330033b"
server_key_iv: "95cbcf1ac8c7e78d59f06006bde2bfdf"
server_key_version: "encryptionkey@0"
I20260925 01:29:41.387214 563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:41.392892 563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:41.393745 563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:41.394030 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:41.394933 563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:41.396330 563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:41.396456 563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:41.396584 563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:41.396675 563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:41.432300 563 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:44875
I20260925 01:29:41.432353 689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:44875 every 8 connection(s)
I20260925 01:29:41.433943 563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:41.435454 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 563
I20260925 01:29:41.435681 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:41.436338 28288 external_mini_cluster.cc:1428] Setting key 1ed09d3bf7d04fa03ad971f8e91a2911
I20260925 01:29:41.439538 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40233
--builtin_ntp_servers=127.27.160.20:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 01:29:41.443585 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:41.436934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:47909 (local address 127.27.160.62:40233)
0925 01:29:41.437468 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:41.437472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:41.439126 (+ 1654us) server_negotiation.cc:407] Connection header received
0925 01:29:41.441077 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:41.441086 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:41.441182 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:41.441304 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:41.442709 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:41.442717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:41.442725 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:41.442748 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:41.442803 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:41.442822 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:41.442955 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:41.443138 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 01:29:41.443205 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":304,"threads_started":1}
I20260925 01:29:41.444402 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:41.437750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40233 (local address 127.27.160.1:47909)
0925 01:29:41.438958 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:41.439006 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:41.440676 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:41.441558 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:41.441566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:41.441667 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:41.442394 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:29:41.442397 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:41.442435 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:29:41.442527 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:41.442925 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:41.442932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:41.442938 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 01:29:41.443172 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 01:29:41.443380 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":376,"threads_started":1}
I20260925 01:29:41.446020 690 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40233
I20260925 01:29:41.446343 690 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:41.447021 690 heartbeater.cc:511] Master 127.27.160.62:40233 requested a full tablet report, sending...
I20260925 01:29:41.449303 504 ts_manager.cc:194] Registered new tserver with Master: 5222a08177f04b1eb79754e9f387528e (127.27.160.1:44875)
W20260925 01:29:41.766911 694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:41.767262 694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:41.767437 694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:41.774824 694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:41.774977 694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:41.775168 694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:41.784310 694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:41.785674 694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:41.787464 694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:41.801673 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:41.801781 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:41.801707 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:41.802979 694 server_base.cc:1034] running on GCE node
I20260925 01:29:41.803650 694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:41.804636 694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:41.805868 694 hybrid_clock.cc:648] HybridClock initialized: now 1790299781805790 us; error 77 us; skew 500 ppm
I20260925 01:29:41.806203 694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:41.808591 694 webserver.cc:460] Webserver started at http://127.27.160.2:45385/ using document root <none> and password file <none>
I20260925 01:29:41.809406 694 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:41.809545 694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:41.809906 694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:41.812711 694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "dbb8de22b1fc49e6a17c8692803c19b4"
format_stamp: "Formatted at 2026-09-25 01:29:41 on dist-test-slave-h6hf"
server_key: "c2a2d80b564a6615840346f0c9d26b11"
server_key_iv: "db22fa629299d2f59dc327178ea17fb0"
server_key_version: "encryptionkey@0"
I20260925 01:29:41.813684 694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "dbb8de22b1fc49e6a17c8692803c19b4"
format_stamp: "Formatted at 2026-09-25 01:29:41 on dist-test-slave-h6hf"
server_key: "c2a2d80b564a6615840346f0c9d26b11"
server_key_iv: "db22fa629299d2f59dc327178ea17fb0"
server_key_version: "encryptionkey@0"
I20260925 01:29:41.819146 694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 01:29:41.822782 708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:41.824410 694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:29:41.824575 694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "dbb8de22b1fc49e6a17c8692803c19b4"
format_stamp: "Formatted at 2026-09-25 01:29:41 on dist-test-slave-h6hf"
server_key: "c2a2d80b564a6615840346f0c9d26b11"
server_key_iv: "db22fa629299d2f59dc327178ea17fb0"
server_key_version: "encryptionkey@0"
I20260925 01:29:41.824793 694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:41.831319 694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:41.832209 694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:41.832479 694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:41.833585 694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:41.835163 694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:41.835268 694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:41.835367 694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:41.835438 694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:41.872997 694 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:45329
I20260925 01:29:41.873059 820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:45329 every 8 connection(s)
I20260925 01:29:41.874616 694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:41.881659 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 694
I20260925 01:29:41.881891 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:41.882625 28288 external_mini_cluster.cc:1428] Setting key e888f2217c604c3fae296cdae3f8413b
I20260925 01:29:41.883427 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:41.877328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:52377 (local address 127.27.160.62:40233)
0925 01:29:41.877564 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:41.877570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:41.879099 (+ 1529us) server_negotiation.cc:407] Connection header received
0925 01:29:41.880811 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:41.880817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:41.880888 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:41.881012 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:41.882613 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:41.882618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:41.882621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:41.882639 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:41.882673 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:41.882685 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:41.882835 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:41.883063 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 01:29:41.883138 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 01:29:41.884359 823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:41.877946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40233 (local address 127.27.160.2:52377)
0925 01:29:41.878887 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:41.878941 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:41.880515 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:41.881290 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:41.881299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:41.881398 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:41.882198 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:29:41.882202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:41.882263 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:29:41.882410 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:41.882800 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:41.882811 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:41.882819 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:29:41.883127 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 01:29:41.883400 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":258,"threads_started":1}
I20260925 01:29:41.886059 821 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40233
I20260925 01:29:41.886125 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40233
--builtin_ntp_servers=127.27.160.20:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 01:29:41.886370 821 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:41.886941 821 heartbeater.cc:511] Master 127.27.160.62:40233 requested a full tablet report, sending...
I20260925 01:29:41.888327 504 ts_manager.cc:194] Registered new tserver with Master: dbb8de22b1fc49e6a17c8692803c19b4 (127.27.160.2:45329)
W20260925 01:29:42.228066 824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:42.228412 824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:42.228595 824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:42.236662 824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:42.236766 824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:42.236984 824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:42.246469 824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:42.247725 824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:42.249630 824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:42.262997 830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:42.263075 829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:42.263078 832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:42.263773 824 server_base.cc:1034] running on GCE node
I20260925 01:29:42.264397 824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:42.265405 824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:42.266660 824 hybrid_clock.cc:648] HybridClock initialized: now 1790299782266612 us; error 72 us; skew 500 ppm
I20260925 01:29:42.267020 824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:42.269449 824 webserver.cc:460] Webserver started at http://127.27.160.3:46035/ using document root <none> and password file <none>
I20260925 01:29:42.270252 824 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:42.270385 824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:42.270714 824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:42.273700 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "1e0bd03cd3054695b5c4f149851ddcaa"
format_stamp: "Formatted at 2026-09-25 01:29:42 on dist-test-slave-h6hf"
server_key: "6ded744312926d7de9651bc07af2ca63"
server_key_iv: "3f4ce5a012b2a809d7d2f606afb095f2"
server_key_version: "encryptionkey@0"
I20260925 01:29:42.274516 824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "1e0bd03cd3054695b5c4f149851ddcaa"
format_stamp: "Formatted at 2026-09-25 01:29:42 on dist-test-slave-h6hf"
server_key: "6ded744312926d7de9651bc07af2ca63"
server_key_iv: "3f4ce5a012b2a809d7d2f606afb095f2"
server_key_version: "encryptionkey@0"
I20260925 01:29:42.279852 824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:29:42.283596 838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:42.284969 824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 01:29:42.285202 824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "1e0bd03cd3054695b5c4f149851ddcaa"
format_stamp: "Formatted at 2026-09-25 01:29:42 on dist-test-slave-h6hf"
server_key: "6ded744312926d7de9651bc07af2ca63"
server_key_iv: "3f4ce5a012b2a809d7d2f606afb095f2"
server_key_version: "encryptionkey@0"
I20260925 01:29:42.285437 824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:42.291326 824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:42.292263 824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:42.292536 824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:42.293499 824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:42.294919 824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:42.295049 824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:42.295183 824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:42.295269 824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:42.328078 824 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:40933
I20260925 01:29:42.328128 950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:40933 every 8 connection(s)
I20260925 01:29:42.329551 824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:42.336836 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 824
I20260925 01:29:42.337014 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:42.337479 28288 external_mini_cluster.cc:1428] Setting key 47c75e6938b84757c34f31ea50d8e049
I20260925 01:29:42.338649 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.332295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:34315 (local address 127.27.160.62:40233)
0925 01:29:42.332476 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.332480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.333986 (+ 1506us) server_negotiation.cc:407] Connection header received
0925 01:29:42.335798 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.335804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.335885 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.335998 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.337520 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.337533 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.337536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.337553 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.337586 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.337598 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.337745 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.338202 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.338284 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:29:42.339564 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.332750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40233 (local address 127.27.160.3:34315)
0925 01:29:42.333823 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.333865 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:42.335522 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:42.336247 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:42.336255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:42.336393 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:42.337210 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:29:42.337212 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:42.337251 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:29:42.337336 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:42.337717 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:42.337729 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:42.337739 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:29:42.338143 (+ 404us) client_negotiation.cc:240] Negotiation successful
0925 01:29:42.338351 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":379,"threads_started":1}
I20260925 01:29:42.341288 951 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40233
I20260925 01:29:42.341594 951 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:42.342181 951 heartbeater.cc:511] Master 127.27.160.62:40233 requested a full tablet report, sending...
I20260925 01:29:42.343537 504 ts_manager.cc:194] Registered new tserver with Master: 1e0bd03cd3054695b5c4f149851ddcaa (127.27.160.3:40933)
I20260925 01:29:42.353694 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:29:42.361053 692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.358528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.27.160.62:40233)
0925 01:29:42.358711 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.358716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.359262 (+ 546us) server_negotiation.cc:407] Connection header received
0925 01:29:42.359376 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.359380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.359443 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.359550 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.360487 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.360492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.360495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.360510 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.360542 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.360553 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.360652 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.360782 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.360830 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:29:42.364921 504 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:42.372277 504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:42.414714 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.385242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.0.0.1:41248)
0925 01:29:42.402246 (+ 17004us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.402290 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:42.402470 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:42.412541 (+ 10071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:42.412547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:42.412631 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:42.413508 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:29:42.413511 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:42.413550 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:29:42.413635 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:42.414139 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:42.414148 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:42.414153 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:29:42.414283 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 01:29:42.414357 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16722,"thread_start_us":16546,"threads_started":1}
I20260925 01:29:42.415014 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.385589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.27.160.1:44875)
0925 01:29:42.411519 (+ 25930us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.411533 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.411580 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:29:42.411686 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.411701 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.412200 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.412443 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.413824 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.413839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.413873 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.413909 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.413985 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.414015 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.414212 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.414305 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.414435 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25679,"thread_start_us":25499,"threads_started":1}
I20260925 01:29:42.418674 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.386219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:40933 (local address 127.0.0.1:53554)
0925 01:29:42.414989 (+ 28770us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.415007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:42.415093 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:42.416850 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:42.416854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:42.416887 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:42.417544 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:29:42.417547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:42.417565 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:29:42.417615 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:42.418117 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:42.418123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:42.418130 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:29:42.418252 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 01:29:42.418317 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28675,"thread_start_us":24452,"threads_started":1}
I20260925 01:29:42.419204 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.385768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:45329 (local address 127.0.0.1:46104)
0925 01:29:42.410343 (+ 24575us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.410373 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:42.410551 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:42.417144 (+ 6593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:42.417153 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:42.417186 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:42.418052 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:29:42.418055 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:42.418069 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:29:42.418118 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:42.418600 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:42.418605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:42.418609 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 01:29:42.418756 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 01:29:42.418809 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24363,"thread_start_us":29634,"threads_started":1}
I20260925 01:29:42.419417 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.387028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.27.160.3:40933)
0925 01:29:42.409473 (+ 22445us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.409488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.415226 (+ 5738us) server_negotiation.cc:407] Connection header received
0925 01:29:42.415309 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.415321 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.415590 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.415747 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.417800 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.417813 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.417831 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.417869 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.417942 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.417970 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.418214 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.418296 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.418417 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22162,"thread_start_us":22477,"threads_started":1}
I20260925 01:29:42.419804 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.386401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.27.160.2:45329)
0925 01:29:42.416046 (+ 29645us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.416056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.416090 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:29:42.416165 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.416175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.416399 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.416578 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.418297 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.418305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.418318 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.418344 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.418398 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.418415 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.418544 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.419412 (+ 868us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.419507 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29453,"thread_start_us":30426,"threads_started":1}
I20260925 01:29:42.420153 625 tablet_service.cc:1467] Processing CreateTablet for tablet f839e6060718420a97ed0505a74c5638 (DEFAULT_TABLE table=test-table [id=7ed5e84e04124990b8b2aff74a3e95f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:42.421703 625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f839e6060718420a97ed0505a74c5638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:42.423390 886 tablet_service.cc:1467] Processing CreateTablet for tablet f839e6060718420a97ed0505a74c5638 (DEFAULT_TABLE table=test-table [id=7ed5e84e04124990b8b2aff74a3e95f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:42.423658 756 tablet_service.cc:1467] Processing CreateTablet for tablet f839e6060718420a97ed0505a74c5638 (DEFAULT_TABLE table=test-table [id=7ed5e84e04124990b8b2aff74a3e95f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:42.424965 886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f839e6060718420a97ed0505a74c5638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:42.425024 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f839e6060718420a97ed0505a74c5638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:42.443615 971 tablet_bootstrap.cc:492] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: Bootstrap starting.
I20260925 01:29:42.448501 971 tablet_bootstrap.cc:654] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:42.452435 690 heartbeater.cc:503] Master 127.27.160.62:40233 was elected leader, sending a full tablet report...
I20260925 01:29:42.452503 969 tablet_bootstrap.cc:492] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e: Bootstrap starting.
I20260925 01:29:42.455801 971 log.cc:824] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:42.457417 969 tablet_bootstrap.cc:654] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:42.457532 970 tablet_bootstrap.cc:492] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Bootstrap starting.
I20260925 01:29:42.458679 969 log.cc:824] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:42.460124 971 tablet_bootstrap.cc:492] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: No bootstrap required, opened a new log
I20260925 01:29:42.460583 971 ts_tablet_manager.cc:1397] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260925 01:29:42.461449 970 tablet_bootstrap.cc:654] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:42.462656 970 log.cc:824] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:42.463346 969 tablet_bootstrap.cc:492] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e: No bootstrap required, opened a new log
I20260925 01:29:42.463629 969 ts_tablet_manager.cc:1397] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 01:29:42.465847 970 tablet_bootstrap.cc:492] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: No bootstrap required, opened a new log
I20260925 01:29:42.466110 970 ts_tablet_manager.cc:1397] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 01:29:42.470216 971 raft_consensus.cc:348] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.470674 971 raft_consensus.cc:374] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:42.470814 971 raft_consensus.cc:729] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb8de22b1fc49e6a17c8692803c19b4, State: Initialized, Role: FOLLOWER
I20260925 01:29:42.471580 971 consensus_queue.cc:260] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.473035 821 heartbeater.cc:503] Master 127.27.160.62:40233 was elected leader, sending a full tablet report...
I20260925 01:29:42.473665 971 ts_tablet_manager.cc:1428] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 01:29:42.474119 969 raft_consensus.cc:348] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.474541 969 raft_consensus.cc:374] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:42.474649 969 raft_consensus.cc:729] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5222a08177f04b1eb79754e9f387528e, State: Initialized, Role: FOLLOWER
I20260925 01:29:42.475291 969 consensus_queue.cc:260] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.477048 969 ts_tablet_manager.cc:1428] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 01:29:42.477952 970 raft_consensus.cc:348] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.478443 970 raft_consensus.cc:374] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:42.478547 970 raft_consensus.cc:729] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0bd03cd3054695b5c4f149851ddcaa, State: Initialized, Role: FOLLOWER
I20260925 01:29:42.479331 970 consensus_queue.cc:260] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.480602 951 heartbeater.cc:503] Master 127.27.160.62:40233 was elected leader, sending a full tablet report...
I20260925 01:29:42.481231 970 ts_tablet_manager.cc:1428] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260925 01:29:42.542574 977 raft_consensus.cc:482] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:42.542865 977 raft_consensus.cc:504] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.544556 977 leader_election.cc:288] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5222a08177f04b1eb79754e9f387528e (127.27.160.1:44875), dbb8de22b1fc49e6a17c8692803c19b4 (127.27.160.2:45329)
I20260925 01:29:42.547607 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.545157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.27.160.3:53245)
0925 01:29:42.545399 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.545419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:42.545583 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:42.545971 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:42.545976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:42.546001 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:42.546682 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:29:42.546684 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:42.546699 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:29:42.546739 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:42.547197 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:42.547205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:42.547212 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:29:42.547316 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:29:42.547360 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260925 01:29:42.547699 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.545195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:53245 (local address 127.27.160.1:44875)
0925 01:29:42.545380 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.545387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.545625 (+ 238us) server_negotiation.cc:407] Connection header received
0925 01:29:42.545714 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.545721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.545790 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.545907 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.546894 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.546900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.546905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.546936 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.546965 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.546976 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.547107 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.547349 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.547408 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:29:42.548316 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.545200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:52461 (local address 127.27.160.2:45329)
0925 01:29:42.545541 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.545552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.546080 (+ 528us) server_negotiation.cc:407] Connection header received
0925 01:29:42.546253 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.546261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.546325 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.546432 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.547554 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.547559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.547563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.547581 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.547619 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.547636 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.547751 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.547980 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.548041 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260925 01:29:42.548447 978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.545206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:45329 (local address 127.27.160.3:52461)
0925 01:29:42.545976 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.545991 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:42.546093 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:42.546509 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:42.546514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:42.546541 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:42.547338 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:29:42.547341 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:42.547358 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:29:42.547399 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:42.547805 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:42.547810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:42.547815 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:29:42.547988 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 01:29:42.548057 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":422,"threads_started":1}
I20260925 01:29:42.548755 645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f839e6060718420a97ed0505a74c5638" candidate_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5222a08177f04b1eb79754e9f387528e" is_pre_election: true
I20260925 01:29:42.549301 645 raft_consensus.cc:2454] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e0bd03cd3054695b5c4f149851ddcaa in term 0.
I20260925 01:29:42.549382 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f839e6060718420a97ed0505a74c5638" candidate_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" is_pre_election: true
I20260925 01:29:42.549894 776 raft_consensus.cc:2454] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e0bd03cd3054695b5c4f149851ddcaa in term 0.
I20260925 01:29:42.549998 840 leader_election.cc:302] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e0bd03cd3054695b5c4f149851ddcaa, 5222a08177f04b1eb79754e9f387528e; no voters:
I20260925 01:29:42.550406 977 raft_consensus.cc:2781] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:42.550536 977 raft_consensus.cc:482] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:42.550658 977 raft_consensus.cc:3037] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:42.552822 977 raft_consensus.cc:504] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.553508 977 leader_election.cc:288] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [CANDIDATE]: Term 1 election: Requested vote from peers 5222a08177f04b1eb79754e9f387528e (127.27.160.1:44875), dbb8de22b1fc49e6a17c8692803c19b4 (127.27.160.2:45329)
I20260925 01:29:42.553932 645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f839e6060718420a97ed0505a74c5638" candidate_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5222a08177f04b1eb79754e9f387528e"
I20260925 01:29:42.553994 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f839e6060718420a97ed0505a74c5638" candidate_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb8de22b1fc49e6a17c8692803c19b4"
I20260925 01:29:42.554154 645 raft_consensus.cc:3037] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:42.554174 776 raft_consensus.cc:3037] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:42.557688 645 raft_consensus.cc:2454] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0bd03cd3054695b5c4f149851ddcaa in term 1.
I20260925 01:29:42.557724 776 raft_consensus.cc:2454] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e0bd03cd3054695b5c4f149851ddcaa in term 1.
I20260925 01:29:42.558127 839 leader_election.cc:302] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e0bd03cd3054695b5c4f149851ddcaa, dbb8de22b1fc49e6a17c8692803c19b4; no voters:
I20260925 01:29:42.558475 977 raft_consensus.cc:2781] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:42.558915 977 raft_consensus.cc:686] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 1 LEADER]: Becoming Leader. State: Replica: 1e0bd03cd3054695b5c4f149851ddcaa, State: Running, Role: LEADER
I20260925 01:29:42.559398 977 consensus_queue.cc:237] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } }
I20260925 01:29:42.564383 504 catalog_manager.cc:5495] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e0bd03cd3054695b5c4f149851ddcaa (127.27.160.3). New cstate: current_term: 1 leader_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e0bd03cd3054695b5c4f149851ddcaa" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40933 } health_report { overall_health: HEALTHY } } }
W20260925 01:29:42.581447 952 tablet.cc:2369] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:42.593453 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.591190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.27.160.3:40933)
0925 01:29:42.591374 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.591380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.591482 (+ 102us) server_negotiation.cc:407] Connection header received
0925 01:29:42.591720 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.591725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.591804 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.591915 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.592796 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.592802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.592807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.592827 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.592863 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.592876 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.593029 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.593133 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.593195 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:29:42.602154 645 raft_consensus.cc:1261] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e [term 1 FOLLOWER]: Refusing update from remote peer 1e0bd03cd3054695b5c4f149851ddcaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:42.602155 776 raft_consensus.cc:1261] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 1 FOLLOWER]: Refusing update from remote peer 1e0bd03cd3054695b5c4f149851ddcaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:42.603264 979 consensus_queue.cc:1035] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5222a08177f04b1eb79754e9f387528e" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 44875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:42.603627 977 consensus_queue.cc:1035] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb8de22b1fc49e6a17c8692803c19b4" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:42.615087 982 mvcc.cc:205] Tried to move back new op lower bound from 7333067909526704128 to 7333067909367136256. Current Snapshot: MvccSnapshot[applied={T|T < 7333067909526704128}]
I20260925 01:29:42.619755 984 mvcc.cc:205] Tried to move back new op lower bound from 7333067909526704128 to 7333067909367136256. Current Snapshot: MvccSnapshot[applied={T|T < 7333067909526704128}]
I20260925 01:29:42.631244 504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:42.631531 504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:42.635226 504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87796fc161e941f784f62e31f24db780: Sending DeleteTablet for 3 replicas of tablet f839e6060718420a97ed0505a74c5638
I20260925 01:29:42.636413 625 tablet_service.cc:1514] Processing DeleteTablet for tablet f839e6060718420a97ed0505a74c5638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:42 UTC) from {username='slave'} at 127.0.0.1:41248
I20260925 01:29:42.636503 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f839e6060718420a97ed0505a74c5638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:42 UTC) from {username='slave'} at 127.0.0.1:46104
I20260925 01:29:42.636722 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f839e6060718420a97ed0505a74c5638 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:42 UTC) from {username='slave'} at 127.0.0.1:53554
I20260925 01:29:42.639015 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 563
I20260925 01:29:42.668255 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.641014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.27.160.1:44875)
0925 01:29:42.641271 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.641281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.664618 (+ 23337us) server_negotiation.cc:407] Connection header received
0925 01:29:42.665331 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.665341 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.665497 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.665607 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.667150 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.667160 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.667168 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.667200 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.667258 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.667280 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.667416 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.667724 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.667812 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:29:42.670184 685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:42.671363 993 tablet_replica.cc:330] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: stopping tablet replica
I20260925 01:29:42.671733 994 tablet_replica.cc:330] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: stopping tablet replica
I20260925 01:29:42.671947 993 raft_consensus.cc:2229] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:42.672245 994 raft_consensus.cc:2229] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:42.672539 993 raft_consensus.cc:2258] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:42.674012 994 raft_consensus.cc:2258] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:42.675541 993 ts_tablet_manager.cc:1905] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:42.677196 994 ts_tablet_manager.cc:1905] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:42.685783 993 ts_tablet_manager.cc:1918] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:42.686141 993 log.cc:1196] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/f839e6060718420a97ed0505a74c5638
I20260925 01:29:42.686859 993 ts_tablet_manager.cc:1939] T f839e6060718420a97ed0505a74c5638 P dbb8de22b1fc49e6a17c8692803c19b4: Deleting consensus metadata
I20260925 01:29:42.689659 488 catalog_manager.cc:4841] TS dbb8de22b1fc49e6a17c8692803c19b4 (127.27.160.2:45329): tablet f839e6060718420a97ed0505a74c5638 (table test-table [id=7ed5e84e04124990b8b2aff74a3e95f4]) successfully deleted
I20260925 01:29:42.691104 994 ts_tablet_manager.cc:1918] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:42.691293 994 log.cc:1196] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/f839e6060718420a97ed0505a74c5638
I20260925 01:29:42.691905 994 ts_tablet_manager.cc:1939] T f839e6060718420a97ed0505a74c5638 P 1e0bd03cd3054695b5c4f149851ddcaa: Deleting consensus metadata
I20260925 01:29:42.693739 491 catalog_manager.cc:4841] TS 1e0bd03cd3054695b5c4f149851ddcaa (127.27.160.3:40933): tablet f839e6060718420a97ed0505a74c5638 (table test-table [id=7ed5e84e04124990b8b2aff74a3e95f4]) successfully deleted
I20260925 01:29:42.774142 992 tablet_replica.cc:330] T f839e6060718420a97ed0505a74c5638 P 5222a08177f04b1eb79754e9f387528e: stopping tablet replica
I20260925 01:29:42.774468 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 563
I20260925 01:29:42.792470 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 694
W20260925 01:29:42.792486 489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.160.1:44875 (error 108)
I20260925 01:29:42.794505 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.794008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.0.0.1:41270)
0925 01:29:42.794203 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.794342 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 01:29:42.794804 489 catalog_manager.cc:4584] TS 5222a08177f04b1eb79754e9f387528e (127.27.160.1:44875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f839e6060718420a97ed0505a74c5638: Network error: Client connection negotiation failed: client connection to 127.27.160.1:44875: connect: Connection refused (error 111)
I20260925 01:29:42.795881 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.793741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.27.160.2:45329)
0925 01:29:42.793918 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.793924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.793955 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:29:42.794022 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.794029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.794144 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.794275 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.795274 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.795287 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.795291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.795306 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.795335 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.795346 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.795473 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.795569 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.795627 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 01:29:42.796600 816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:42.848587 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.847985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.0.0.1:41284)
0925 01:29:42.848247 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.848364 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 01:29:42.849774 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.849358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.0.0.1:41288)
0925 01:29:42.849536 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.849630 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 01:29:42.886940 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 694
I20260925 01:29:42.904856 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 824
I20260925 01:29:42.908536 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.905769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.27.160.3:40933)
0925 01:29:42.905962 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:42.905970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:42.906149 (+ 179us) server_negotiation.cc:407] Connection header received
0925 01:29:42.906429 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:42.906438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:42.906497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:42.906620 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:42.907653 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:42.907660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:42.907665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:42.907702 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:42.907738 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:42.907751 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:42.907924 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:42.908061 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 01:29:42.908126 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:29:42.909597 946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:42.910733 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.909968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.0.0.1:41290)
0925 01:29:42.910321 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.910489 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260925 01:29:42.912072 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.911651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.0.0.1:41306)
0925 01:29:42.911830 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.911930 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 01:29:42.993096 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.992535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.0.0.1:41314)
0925 01:29:42.992795 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.992918 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260925 01:29:42.994308 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:42.993888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:44875 (local address 127.0.0.1:41320)
0925 01:29:42.994124 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:42.994194 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260925 01:29:43.031952 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 824
I20260925 01:29:43.051155 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 473
I20260925 01:29:43.052367 534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:43.162607 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 473
2026-09-25T01:29:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2747 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 01:29:43.204159 28288 mini_oidc.cc:152] Starting JWKS server
I20260925 01:29:43.206804 28288 webserver.cc:460] Webserver started at https://127.0.0.1:36521/ using document root <none> and password file <none>
I20260925 01:29:43.207144 28288 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:29:43.207775 28288 webserver.cc:460] Webserver started at http://127.0.0.1:42055/ using document root <none> and password file <none>
2026-09-25T01:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:43Z Disabled control of system clock
I20260925 01:29:43.235685 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37335
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:40349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:37335
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36521/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/testchainca.cert with env {}
W20260925 01:29:43.580178 1006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:43.580552 1006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:43.580718 1006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:43.588881 1006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:43.589077 1006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:43.589211 1006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:43.589340 1006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:43.589430 1006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:29:43.589532 1006 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:29:43.589619 1006 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36521/jwks.json
I20260925 01:29:43.600117 1006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:37335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36521/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:43.601672 1006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:43.603099 1006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:43.616074 1011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:43.615986 1012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:43.616110 1014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:43.617110 1006 server_base.cc:1034] running on GCE node
I20260925 01:29:43.618122 1006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:43.619639 1006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:43.620944 1006 hybrid_clock.cc:648] HybridClock initialized: now 1790299783620861 us; error 95 us; skew 500 ppm
I20260925 01:29:43.621418 1006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:43.623889 1006 webserver.cc:460] Webserver started at http://127.27.160.62:45915/ using document root <none> and password file <none>
I20260925 01:29:43.624625 1006 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:43.624775 1006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:43.625204 1006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:43.628388 1006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "9af309677a054c59a7f1ae6e939196fc"
format_stamp: "Formatted at 2026-09-25 01:29:43 on dist-test-slave-h6hf"
server_key: "cf76eb7b96e4638248e51af85eced9c8"
server_key_iv: "871f5f20c10650d1e49c57314073d3fb"
server_key_version: "encryptionkey@0"
I20260925 01:29:43.629612 1006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "9af309677a054c59a7f1ae6e939196fc"
format_stamp: "Formatted at 2026-09-25 01:29:43 on dist-test-slave-h6hf"
server_key: "cf76eb7b96e4638248e51af85eced9c8"
server_key_iv: "871f5f20c10650d1e49c57314073d3fb"
server_key_version: "encryptionkey@0"
I20260925 01:29:43.635667 1006 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260925 01:29:43.639780 1020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:43.641433 1006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:29:43.641660 1006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "9af309677a054c59a7f1ae6e939196fc"
format_stamp: "Formatted at 2026-09-25 01:29:43 on dist-test-slave-h6hf"
server_key: "cf76eb7b96e4638248e51af85eced9c8"
server_key_iv: "871f5f20c10650d1e49c57314073d3fb"
server_key_version: "encryptionkey@0"
I20260925 01:29:43.641916 1006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:43.666316 1006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:43.667196 1006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:43.667518 1006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:43.687147 1006 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:37335
I20260925 01:29:43.687206 1071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:37335 every 8 connection(s)
I20260925 01:29:43.688509 1006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:43.692695 1072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:43.696630 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1006
I20260925 01:29:43.696897 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:43.697894 28288 external_mini_cluster.cc:1428] Setting key e55cc151bcce49a862cf30d274e4f3e2
I20260925 01:29:43.707274 1072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc: Bootstrap starting.
I20260925 01:29:43.711726 1072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:43.713199 1072 log.cc:824] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:43.713505 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:43.700216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.27.160.62:37335)
0925 01:29:43.701408 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:43.701419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:43.701463 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:29:43.703165 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:43.703191 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:43.703714 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:43.704180 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:43.704822 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:43.706190 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:43.707619 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:43.708759 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:43.708983 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:43.710230 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:43.710246 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:43.710271 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:43.711073 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:43.711199 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:43.711230 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:43.711359 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:43.711590 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 01:29:43.711988 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":366,"threads_started":1}
I20260925 01:29:43.716795 1072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc: No bootstrap required, opened a new log
I20260925 01:29:43.717945 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:29:43.727044 1072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af309677a054c59a7f1ae6e939196fc" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37335 } }
I20260925 01:29:43.727485 1072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:43.727578 1072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af309677a054c59a7f1ae6e939196fc, State: Initialized, Role: FOLLOWER
I20260925 01:29:43.728315 1072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af309677a054c59a7f1ae6e939196fc" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37335 } }
I20260925 01:29:43.728663 1072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:43.728889 1072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:43.729111 1072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:43.731868 1072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af309677a054c59a7f1ae6e939196fc" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37335 } }
I20260925 01:29:43.732476 1072 leader_election.cc:302] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9af309677a054c59a7f1ae6e939196fc; no voters:
I20260925 01:29:43.733234 1072 leader_election.cc:288] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:43.733335 1077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:43.734333 1077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [term 1 LEADER]: Becoming Leader. State: Replica: 9af309677a054c59a7f1ae6e939196fc, State: Running, Role: LEADER
I20260925 01:29:43.734963 1077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af309677a054c59a7f1ae6e939196fc" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37335 } }
I20260925 01:29:43.735929 1072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:43.738605 1078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9af309677a054c59a7f1ae6e939196fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af309677a054c59a7f1ae6e939196fc" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37335 } } }
I20260925 01:29:43.739089 1078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:43.739642 1079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9af309677a054c59a7f1ae6e939196fc. Latest consensus state: current_term: 1 leader_uuid: "9af309677a054c59a7f1ae6e939196fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9af309677a054c59a7f1ae6e939196fc" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37335 } } }
I20260925 01:29:43.739987 1079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:43.741108 1085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:43.747690 1085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:43.759966 1085 catalog_manager.cc:1347] Generated new cluster ID: 922a0f8d26cc410d9b5708f327cd9354
I20260925 01:29:43.760108 1085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:43.792526 1085 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:43.793908 1085 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:43.808106 1085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9af309677a054c59a7f1ae6e939196fc: Generated new TSK 0
I20260925 01:29:43.809554 1085 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 01:29:43.857807 1102 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 01:29:43.858033 1102 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 01:29:43.858951 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:43.828500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.27.160.62:37335)
0925 01:29:43.828686 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:43.828690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:43.829297 (+ 607us) server_negotiation.cc:407] Connection header received
0925 01:29:43.829647 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:43.829651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:43.829722 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:43.829878 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:29:43.830416 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:43.831148 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:43.832578 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:43.833040 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:43.833229 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:43.833742 (+ 513us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:29:43.858199 (+ 24457us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 01:29:43.858615 (+ 416us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 01:29:43.859102 1075 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 01:29:43.860734 1101 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:29:43.828484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37335 (local address 127.0.0.1:41768)
0925 01:29:43.829199 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:43.829215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:43.829402 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:43.829902 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:43.829907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:43.829920 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:29:43.830259 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:43.830268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:43.831303 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:43.831308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:43.832425 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:43.832436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:43.833238 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:43.833244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:43.833397 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:43.833459 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:29:43.858562 (+ 25103us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 01:29:43.858988 (+ 426us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.160.62:37335: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":360,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.160.62:37335: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36521/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 01:29:43.862666 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1006
I20260925 01:29:43.863839 1067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:43.958653 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1006
2026-09-25T01:29:43Z chronyd exiting
I20260925 01:29:44.010658 28288 test_util.cc:175] -----------------------------------------------
I20260925 01:29:44.010859 28288 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790299737515179-28288-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (811 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 01:29:44.014833 28288 mini_oidc.cc:152] Starting JWKS server
I20260925 01:29:44.016779 28288 webserver.cc:460] Webserver started at https://127.0.0.1:46843/ using document root <none> and password file <none>
I20260925 01:29:44.017053 28288 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:29:44.017829 28288 webserver.cc:460] Webserver started at http://127.0.0.1:41445/ using document root <none> and password file <none>
2026-09-25T01:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:44Z Disabled control of system clock
I20260925 01:29:44.045856 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:41589
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:41589
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46843/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/testchainca.cert with env {}
W20260925 01:29:44.370136 1109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:44.370514 1109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:44.370671 1109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:44.378098 1109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:44.378242 1109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:44.378330 1109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:44.378451 1109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:44.378530 1109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:29:44.378625 1109 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:29:44.378702 1109 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46843/jwks.json
I20260925 01:29:44.389214 1109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:41589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:41589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46843/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:44.390569 1109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:44.391901 1109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:44.404057 1114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:44.404088 1117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:44.404139 1115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:44.404948 1109 server_base.cc:1034] running on GCE node
I20260925 01:29:44.405781 1109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:44.407218 1109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:44.408428 1109 hybrid_clock.cc:648] HybridClock initialized: now 1790299784408385 us; error 41 us; skew 500 ppm
I20260925 01:29:44.408813 1109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:44.411401 1109 webserver.cc:460] Webserver started at http://127.27.160.62:35763/ using document root <none> and password file <none>
I20260925 01:29:44.412155 1109 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:44.412252 1109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:44.412637 1109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:44.415756 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5"
format_stamp: "Formatted at 2026-09-25 01:29:44 on dist-test-slave-h6hf"
server_key: "c4e1f5b53cfbe4961e710c0b73b4433a"
server_key_iv: "c0a3845fe7ef5128e14b293ff3909f80"
server_key_version: "encryptionkey@0"
I20260925 01:29:44.416630 1109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5"
format_stamp: "Formatted at 2026-09-25 01:29:44 on dist-test-slave-h6hf"
server_key: "c4e1f5b53cfbe4961e710c0b73b4433a"
server_key_iv: "c0a3845fe7ef5128e14b293ff3909f80"
server_key_version: "encryptionkey@0"
I20260925 01:29:44.422158 1109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:29:44.425933 1123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:44.427454 1109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:44.427632 1109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5"
format_stamp: "Formatted at 2026-09-25 01:29:44 on dist-test-slave-h6hf"
server_key: "c4e1f5b53cfbe4961e710c0b73b4433a"
server_key_iv: "c0a3845fe7ef5128e14b293ff3909f80"
server_key_version: "encryptionkey@0"
I20260925 01:29:44.427837 1109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:44.450160 1109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:44.451035 1109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:44.451349 1109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:44.470342 1109 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:41589
I20260925 01:29:44.470373 1174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:41589 every 8 connection(s)
I20260925 01:29:44.471711 1109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:44.474761 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1109
I20260925 01:29:44.474987 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:44.475716 28288 external_mini_cluster.cc:1428] Setting key eecbdf9f16d1cebc345b2621599e6910
I20260925 01:29:44.476014 1175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:44.491272 1175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5: Bootstrap starting.
I20260925 01:29:44.491531 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:44.477716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.27.160.62:41589)
0925 01:29:44.478808 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:44.478827 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:44.478881 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:29:44.480942 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:44.480974 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:44.481637 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:44.482097 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:44.482698 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:44.484681 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:44.486200 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:44.487268 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:44.487482 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:44.488662 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:44.488673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:44.488695 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:44.489451 (+ 756us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:44.489547 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:44.489573 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:44.489686 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:44.489915 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 01:29:44.490242 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":340,"threads_started":1}
I20260925 01:29:44.494747 1175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:44.495360 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:29:44.495874 1175 log.cc:824] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:44.499033 1175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5: No bootstrap required, opened a new log
I20260925 01:29:44.506551 1175 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41589 } }
I20260925 01:29:44.506901 1175 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:44.506985 1175 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e4b05a20c84df3ae0ec288f16fd6b5, State: Initialized, Role: FOLLOWER
I20260925 01:29:44.507484 1175 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41589 } }
I20260925 01:29:44.507683 1175 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:44.507783 1175 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:44.507911 1175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:44.510092 1175 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41589 } }
I20260925 01:29:44.510603 1175 leader_election.cc:302] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3e4b05a20c84df3ae0ec288f16fd6b5; no voters:
I20260925 01:29:44.511198 1175 leader_election.cc:288] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:44.511366 1180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:44.513526 1175 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:44.513458 1180 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [term 1 LEADER]: Becoming Leader. State: Replica: d3e4b05a20c84df3ae0ec288f16fd6b5, State: Running, Role: LEADER
I20260925 01:29:44.514111 1180 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41589 } }
I20260925 01:29:44.517640 1182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3e4b05a20c84df3ae0ec288f16fd6b5. Latest consensus state: current_term: 1 leader_uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41589 } } }
I20260925 01:29:44.517633 1181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e4b05a20c84df3ae0ec288f16fd6b5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41589 } } }
I20260925 01:29:44.518121 1181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:44.518121 1182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:44.519086 1189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:44.527148 1189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:44.537490 1189 catalog_manager.cc:1347] Generated new cluster ID: c2ac185dd53a4d1c9356b47fe4e8bd08
I20260925 01:29:44.537647 1189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:44.572168 1189 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:44.573625 1189 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:44.581619 1189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3e4b05a20c84df3ae0ec288f16fd6b5: Generated new TSK 0
I20260925 01:29:44.582497 1189 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 01:29:44.622077 1205 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 01:29:44.622224 1205 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 01:29:44.623548 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:44.602663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.27.160.62:41589)
0925 01:29:44.602860 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:44.602866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:44.603263 (+ 397us) server_negotiation.cc:407] Connection header received
0925 01:29:44.603496 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:44.603503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:44.603594 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:44.603773 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:29:44.604223 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:44.605373 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:44.606722 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:44.607432 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:44.607582 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:44.608053 (+ 471us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:29:44.622824 (+ 14771us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 01:29:44.623210 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 01:29:44.623577 1204 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:29:44.602510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41589 (local address 127.0.0.1:57634)
0925 01:29:44.603161 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:44.603177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:44.603306 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:44.603785 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:44.603789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:44.603798 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:29:44.604062 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:44.604070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:44.605534 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:44.605538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:44.606577 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:44.606588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:44.607621 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:44.607625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:44.607755 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:44.607786 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:29:44.623085 (+ 15299us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 01:29:44.623307 (+ 222us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.160.62:41589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":350,"threads_started":1}
W20260925 01:29:44.623679 1178 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57634: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46843/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 01:29:44.624722 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1109
I20260925 01:29:44.626041 1170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:44.722324 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1109
2026-09-25T01:29:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (763 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 01:29:44.777799 28288 mini_oidc.cc:152] Starting JWKS server
I20260925 01:29:44.779685 28288 webserver.cc:460] Webserver started at https://127.0.0.1:41685/ using document root <none> and password file <none>
I20260925 01:29:44.779913 28288 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:29:44.780474 28288 webserver.cc:460] Webserver started at http://127.0.0.1:41429/ using document root <none> and password file <none>
2026-09-25T01:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:44Z Disabled control of system clock
I20260925 01:29:44.808280 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:36747
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:38005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:36747
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41685/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 01:29:45.134068 1212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:45.134433 1212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:45.134608 1212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:45.141918 1212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:45.142084 1212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:45.142158 1212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:45.142244 1212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:45.142334 1212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:29:45.142431 1212 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:29:45.142511 1212 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41685/jwks.json
I20260925 01:29:45.151680 1212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:36747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:36747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41685/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:45.153048 1212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:45.154956 1212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:45.166988 1218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:45.167003 1217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:45.166925 1220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:45.167850 1212 server_base.cc:1034] running on GCE node
I20260925 01:29:45.168614 1212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:45.170142 1212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:45.171343 1212 hybrid_clock.cc:648] HybridClock initialized: now 1790299785171284 us; error 54 us; skew 500 ppm
I20260925 01:29:45.171666 1212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:45.174150 1212 webserver.cc:460] Webserver started at http://127.27.160.62:45795/ using document root <none> and password file <none>
I20260925 01:29:45.174877 1212 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:45.174969 1212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:45.175244 1212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:45.178200 1212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "5aa87eae33c24ee4b0dd29fb826064f3"
format_stamp: "Formatted at 2026-09-25 01:29:45 on dist-test-slave-h6hf"
server_key: "bd0825cb981bb7d168da6a508654abc7"
server_key_iv: "ecae72a4f6eee354016fb00a5b30c21b"
server_key_version: "encryptionkey@0"
I20260925 01:29:45.178980 1212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "5aa87eae33c24ee4b0dd29fb826064f3"
format_stamp: "Formatted at 2026-09-25 01:29:45 on dist-test-slave-h6hf"
server_key: "bd0825cb981bb7d168da6a508654abc7"
server_key_iv: "ecae72a4f6eee354016fb00a5b30c21b"
server_key_version: "encryptionkey@0"
I20260925 01:29:45.184134 1212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:29:45.187573 1226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:45.188942 1212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:29:45.189087 1212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "5aa87eae33c24ee4b0dd29fb826064f3"
format_stamp: "Formatted at 2026-09-25 01:29:45 on dist-test-slave-h6hf"
server_key: "bd0825cb981bb7d168da6a508654abc7"
server_key_iv: "ecae72a4f6eee354016fb00a5b30c21b"
server_key_version: "encryptionkey@0"
I20260925 01:29:45.189328 1212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:45.204444 1212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:45.205294 1212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:45.205554 1212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:45.225032 1212 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:36747
I20260925 01:29:45.225060 1277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:36747 every 8 connection(s)
I20260925 01:29:45.226948 1212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:45.228658 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1212
I20260925 01:29:45.228843 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:45.229628 28288 external_mini_cluster.cc:1428] Setting key 97220fe1b2319dfb42f0407aac7e81ed
I20260925 01:29:45.232802 1278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:45.244513 1278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3: Bootstrap starting.
I20260925 01:29:45.248322 1278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:45.249073 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:45.231681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.27.160.62:36747)
0925 01:29:45.233165 (+ 1484us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:45.233188 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:45.237826 (+ 4638us) server_negotiation.cc:407] Connection header received
0925 01:29:45.239653 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:45.239675 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:45.240153 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:45.240477 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:45.241056 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:45.242470 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:45.244326 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:45.245113 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:45.245328 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:45.246661 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:45.246671 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:45.246686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:45.247234 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:45.247328 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:45.247348 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:45.247436 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:45.247685 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 01:29:45.248017 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":467,"threads_started":1}
I20260925 01:29:45.249615 1278 log.cc:824] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:45.252732 1278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3: No bootstrap required, opened a new log
I20260925 01:29:45.253253 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:29:45.260397 1278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa87eae33c24ee4b0dd29fb826064f3" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36747 } }
I20260925 01:29:45.260881 1278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:45.261000 1278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa87eae33c24ee4b0dd29fb826064f3, State: Initialized, Role: FOLLOWER
I20260925 01:29:45.261608 1278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa87eae33c24ee4b0dd29fb826064f3" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36747 } }
I20260925 01:29:45.261850 1278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:45.261996 1278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:45.262178 1278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:45.264331 1278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa87eae33c24ee4b0dd29fb826064f3" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36747 } }
I20260925 01:29:45.264854 1278 leader_election.cc:302] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aa87eae33c24ee4b0dd29fb826064f3; no voters:
I20260925 01:29:45.265609 1278 leader_election.cc:288] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:45.265810 1283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:45.266597 1283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa87eae33c24ee4b0dd29fb826064f3, State: Running, Role: LEADER
I20260925 01:29:45.267184 1283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa87eae33c24ee4b0dd29fb826064f3" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36747 } }
I20260925 01:29:45.267947 1278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:45.270195 1285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aa87eae33c24ee4b0dd29fb826064f3. Latest consensus state: current_term: 1 leader_uuid: "5aa87eae33c24ee4b0dd29fb826064f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa87eae33c24ee4b0dd29fb826064f3" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36747 } } }
I20260925 01:29:45.270344 1284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aa87eae33c24ee4b0dd29fb826064f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aa87eae33c24ee4b0dd29fb826064f3" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36747 } } }
I20260925 01:29:45.270648 1285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:45.270671 1284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:45.277853 1291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:45.283185 1291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:45.291774 1291 catalog_manager.cc:1347] Generated new cluster ID: 7099d20c39214fd6bbf774e759e3d36e
I20260925 01:29:45.291898 1291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:45.334276 1291 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:45.335636 1291 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:45.346710 1291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aa87eae33c24ee4b0dd29fb826064f3: Generated new TSK 0
I20260925 01:29:45.347638 1291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:29:45.376796 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:45.367936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.27.160.62:36747)
0925 01:29:45.368147 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:45.368154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:45.368659 (+ 505us) server_negotiation.cc:407] Connection header received
0925 01:29:45.368954 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:45.368960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:45.369053 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:45.369261 (+ 208us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:29:45.369739 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:45.370866 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:45.372305 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:45.372983 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:45.373151 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:45.373671 (+ 520us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:29:45.376113 (+ 2442us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 01:29:45.376462 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 01:29:45.376936 1281 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48922: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 01:29:45.378527 1307 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:29:45.367906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:36747 (local address 127.0.0.1:48922)
0925 01:29:45.368553 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:45.368570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:45.368727 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:45.369273 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:45.369276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:45.369286 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:29:45.369579 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:45.369589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:45.370998 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:45.371001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:45.372172 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:45.372184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:45.373218 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:45.373224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:45.373351 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:45.373385 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:29:45.376347 (+ 2962us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 01:29:45.378116 (+ 1769us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.160.62:36747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41685/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":1430,"thread_start_us":369,"threads_started":1}
I20260925 01:29:45.379894 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1212
I20260925 01:29:45.381210 1273 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 01:29:45.389423 1308 webserver.cc:533] Webserver: sslize failed:
E20260925 01:29:45.389550 1308 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 01:29:45.478684 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1212
2026-09-25T01:29:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (756 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T01:29:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:45Z Disabled control of system clock
I20260925 01:29:45.566566 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40169
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:46427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:40169
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/testchainca.cert with env {}
W20260925 01:29:45.887435 1313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:45.887810 1313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:45.887961 1313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:45.895156 1313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:45.895310 1313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:45.895426 1313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:45.895534 1313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:45.895615 1313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:29:45.895704 1313 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:29:45.895779 1313 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 01:29:45.905786 1313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:40169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:45.907176 1313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:45.908553 1313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:45.920784 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:45.920760 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:45.920814 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:45.921845 1313 server_base.cc:1034] running on GCE node
I20260925 01:29:45.922647 1313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:45.924103 1313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:45.925321 1313 hybrid_clock.cc:648] HybridClock initialized: now 1790299785925262 us; error 59 us; skew 500 ppm
I20260925 01:29:45.925680 1313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:45.931376 1313 webserver.cc:460] Webserver started at http://127.27.160.62:36663/ using document root <none> and password file <none>
I20260925 01:29:45.932163 1313 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:45.932314 1313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:45.932636 1313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:45.935705 1313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "f400e292eb114224adcd7d8fe299a7a8"
format_stamp: "Formatted at 2026-09-25 01:29:45 on dist-test-slave-h6hf"
server_key: "771e75bda1d2759f132f42e0f25f83f4"
server_key_iv: "f67efde3861c42ec9ae1dde74a2b43d1"
server_key_version: "encryptionkey@0"
I20260925 01:29:45.936539 1313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "f400e292eb114224adcd7d8fe299a7a8"
format_stamp: "Formatted at 2026-09-25 01:29:45 on dist-test-slave-h6hf"
server_key: "771e75bda1d2759f132f42e0f25f83f4"
server_key_iv: "f67efde3861c42ec9ae1dde74a2b43d1"
server_key_version: "encryptionkey@0"
I20260925 01:29:45.941799 1313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 01:29:45.945446 1327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:45.946820 1313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:45.947018 1313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "f400e292eb114224adcd7d8fe299a7a8"
format_stamp: "Formatted at 2026-09-25 01:29:45 on dist-test-slave-h6hf"
server_key: "771e75bda1d2759f132f42e0f25f83f4"
server_key_iv: "f67efde3861c42ec9ae1dde74a2b43d1"
server_key_version: "encryptionkey@0"
I20260925 01:29:45.947257 1313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:45.993096 1313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:45.994073 1313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:45.994362 1313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:46.013439 1313 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:40169
I20260925 01:29:46.013446 1378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:40169 every 8 connection(s)
I20260925 01:29:46.014838 1313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:46.016901 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1313
I20260925 01:29:46.017107 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:46.018008 28288 external_mini_cluster.cc:1428] Setting key 5d345f978bf85fb5390568cad875a9de
I20260925 01:29:46.019776 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:46.031178 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:46.019951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.27.160.62:40169)
0925 01:29:46.020957 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:46.020967 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:46.021008 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:29:46.022603 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:46.022625 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:46.023082 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:46.023486 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:46.024065 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:46.025446 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:46.026733 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:46.027422 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:46.027627 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:46.028830 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:46.028837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:46.028855 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:29:46.029486 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:46.029594 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:29:46.029618 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:46.029749 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:46.029975 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 01:29:46.030271 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":364,"threads_started":1}
I20260925 01:29:46.031683 1379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8: Bootstrap starting.
I20260925 01:29:46.034926 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:29:46.035022 1379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:46.036087 1379 log.cc:824] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:46.039116 1379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8: No bootstrap required, opened a new log
I20260925 01:29:46.046746 1379 raft_consensus.cc:348] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e292eb114224adcd7d8fe299a7a8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40169 } }
I20260925 01:29:46.047315 1379 raft_consensus.cc:374] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:46.047441 1379 raft_consensus.cc:729] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f400e292eb114224adcd7d8fe299a7a8, State: Initialized, Role: FOLLOWER
I20260925 01:29:46.048101 1379 consensus_queue.cc:260] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e292eb114224adcd7d8fe299a7a8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40169 } }
I20260925 01:29:46.048357 1379 raft_consensus.cc:388] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:46.048507 1379 raft_consensus.cc:482] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:46.048703 1379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:46.051163 1379 raft_consensus.cc:504] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e292eb114224adcd7d8fe299a7a8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40169 } }
I20260925 01:29:46.051694 1379 leader_election.cc:302] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f400e292eb114224adcd7d8fe299a7a8; no voters:
I20260925 01:29:46.052371 1379 leader_election.cc:288] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:46.052516 1384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:46.053263 1384 raft_consensus.cc:686] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [term 1 LEADER]: Becoming Leader. State: Replica: f400e292eb114224adcd7d8fe299a7a8, State: Running, Role: LEADER
I20260925 01:29:46.053794 1384 consensus_queue.cc:237] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e292eb114224adcd7d8fe299a7a8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40169 } }
I20260925 01:29:46.054517 1379 sys_catalog.cc:564] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:46.056535 1386 sys_catalog.cc:455] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f400e292eb114224adcd7d8fe299a7a8. Latest consensus state: current_term: 1 leader_uuid: "f400e292eb114224adcd7d8fe299a7a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e292eb114224adcd7d8fe299a7a8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40169 } } }
I20260925 01:29:46.056742 1385 sys_catalog.cc:455] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f400e292eb114224adcd7d8fe299a7a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e292eb114224adcd7d8fe299a7a8" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40169 } } }
I20260925 01:29:46.057070 1385 sys_catalog.cc:458] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:46.057070 1386 sys_catalog.cc:458] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:46.059188 1389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:46.064736 1389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:46.073390 1389 catalog_manager.cc:1347] Generated new cluster ID: c28ffa5c7ee84bf988d7a70780bc5b26
I20260925 01:29:46.073527 1389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:46.101347 1389 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:46.103344 1389 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:46.118965 1389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f400e292eb114224adcd7d8fe299a7a8: Generated new TSK 0
I20260925 01:29:46.120151 1389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:29:46.728014 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:46.140500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.27.160.62:40169)
0925 01:29:46.140750 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:46.140756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:46.141175 (+ 419us) server_negotiation.cc:407] Connection header received
0925 01:29:46.141423 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:46.141427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:46.141505 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:46.141634 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:29:46.142166 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:46.142906 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:46.144983 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:46.145531 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:46.145623 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:46.146213 (+ 590us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:29:46.727412 (+581199us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 01:29:46.727795 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57322: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 01:29:46.728175 1382 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57322: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 01:29:46.728330 1408 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:29:46.140383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40169 (local address 127.0.0.1:57322)
0925 01:29:46.141015 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:46.141045 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:46.141226 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:46.141685 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:46.141693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:46.141708 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:29:46.142005 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:46.142017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:46.143066 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:46.143071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:46.144839 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:46.144855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:46.145703 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:46.145709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:46.145893 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:46.145939 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:29:46.727734 (+581795us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 01:29:46.727986 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.160.62:40169: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":312,"threads_started":1}
I20260925 01:29:46.729607 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1313
I20260925 01:29:46.730752 1374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:46.832154 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1313
2026-09-25T01:29:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1334 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (413 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (422 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:47 dist-test-slave-h6hf krb5kdc[1423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:47 dist-test-slave-h6hf krb5kdc[1423](info): set up 2 sockets
Sep 25 01:29:47 dist-test-slave-h6hf krb5kdc[1423](info): commencing operation
krb5kdc: starting...
W20260925 01:29:48.766865 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:48 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:29:48.982656 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:41915
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:41915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:49.315321 1436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:49.315629 1436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:49.315794 1436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:49.323287 1436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:49.323395 1436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:49.323482 1436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:49.323580 1436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:49.323669 1436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:49.333748 1436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:41915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:41915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:49.335146 1436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:49.336501 1436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:49.349215 1441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:49.349269 1442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:49.349248 1444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:49.350560 1436 server_base.cc:1034] running on GCE node
I20260925 01:29:49.351444 1436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:49.353350 1436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:49.354622 1436 hybrid_clock.cc:648] HybridClock initialized: now 1790299789354560 us; error 66 us; skew 500 ppm
I20260925 01:29:49.355046 1436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:49 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:49.358364 1436 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:29:49.359880 1436 webserver.cc:460] Webserver started at http://127.27.160.62:37389/ using document root <none> and password file <none>
I20260925 01:29:49.360634 1436 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:49.360783 1436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:49.361110 1436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:49.364254 1436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "60d2c96fcce544f8aee318a5c10a3621"
format_stamp: "Formatted at 2026-09-25 01:29:49 on dist-test-slave-h6hf"
server_key: "7d122f0faaccf3563835b9a780889651"
server_key_iv: "6a5ef2b1dc014422692566d6c7b99db5"
server_key_version: "encryptionkey@0"
I20260925 01:29:49.365203 1436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "60d2c96fcce544f8aee318a5c10a3621"
format_stamp: "Formatted at 2026-09-25 01:29:49 on dist-test-slave-h6hf"
server_key: "7d122f0faaccf3563835b9a780889651"
server_key_iv: "6a5ef2b1dc014422692566d6c7b99db5"
server_key_version: "encryptionkey@0"
I20260925 01:29:49.370702 1436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:29:49.374651 1451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:49.376186 1436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:49.376428 1436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "60d2c96fcce544f8aee318a5c10a3621"
format_stamp: "Formatted at 2026-09-25 01:29:49 on dist-test-slave-h6hf"
server_key: "7d122f0faaccf3563835b9a780889651"
server_key_iv: "6a5ef2b1dc014422692566d6c7b99db5"
server_key_version: "encryptionkey@0"
I20260925 01:29:49.376668 1436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:49.410760 1436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:49.416011 1436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:49.416358 1436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:49.437261 1436 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:41915
I20260925 01:29:49.437391 1502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:41915 every 8 connection(s)
I20260925 01:29:49.438795 1436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:49.442694 1503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:49.444403 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1436
I20260925 01:29:49.444649 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:49.445425 28288 external_mini_cluster.cc:1428] Setting key 5738052580e6d97c121f938daaa2bc7b
I20260925 01:29:49.454941 1503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621: Bootstrap starting.
I20260925 01:29:49.458988 1503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:49.460390 1503 log.cc:824] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:29:49 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:49.463860 1503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621: No bootstrap required, opened a new log
I20260925 01:29:49.472029 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:49.447684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.27.160.62:41915)
0925 01:29:49.448625 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:49.448638 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:49.448680 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:29:49.450550 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:49.450575 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:49.451025 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:49.451460 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:49.453342 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:49.454830 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:49.456747 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:49.457675 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:49.457827 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:49.462977 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:49.462990 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:49.463013 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:49.463066 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:49.467394 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:49.468119 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:49.468129 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:49.468137 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:49.468352 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:49.468718 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:49.468723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:49.468726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:49.469470 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:49.469791 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:49.470147 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 01:29:49.470619 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":454,"threads_started":1}
I20260925 01:29:49.474668 1503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d2c96fcce544f8aee318a5c10a3621" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41915 } }
I20260925 01:29:49.475099 1503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:49.475224 1503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d2c96fcce544f8aee318a5c10a3621, State: Initialized, Role: FOLLOWER
I20260925 01:29:49.475862 1503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d2c96fcce544f8aee318a5c10a3621" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41915 } }
I20260925 01:29:49.476115 1503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:49.476260 1503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:49.476444 1503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:49.478684 1503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d2c96fcce544f8aee318a5c10a3621" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41915 } }
I20260925 01:29:49.479197 1503 leader_election.cc:302] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60d2c96fcce544f8aee318a5c10a3621; no voters:
I20260925 01:29:49.479869 1503 leader_election.cc:288] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:49.480085 1508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:49.480858 1508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [term 1 LEADER]: Becoming Leader. State: Replica: 60d2c96fcce544f8aee318a5c10a3621, State: Running, Role: LEADER
I20260925 01:29:49.481473 1508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d2c96fcce544f8aee318a5c10a3621" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41915 } }
I20260925 01:29:49.482180 1503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:49.485028 1510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60d2c96fcce544f8aee318a5c10a3621. Latest consensus state: current_term: 1 leader_uuid: "60d2c96fcce544f8aee318a5c10a3621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d2c96fcce544f8aee318a5c10a3621" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41915 } } }
I20260925 01:29:49.485455 1510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:49.485977 1509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60d2c96fcce544f8aee318a5c10a3621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d2c96fcce544f8aee318a5c10a3621" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41915 } } }
I20260925 01:29:49.486254 1509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:49.488013 1517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:49.493993 1517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:49.503901 1517 catalog_manager.cc:1347] Generated new cluster ID: e291aa71d36047a78d7bdfc8948153ba
I20260925 01:29:49.504038 1517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:49.514163 1517 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:49.516405 1517 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:49.524936 1517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60d2c96fcce544f8aee318a5c10a3621: Generated new TSK 0
I20260925 01:29:49.526218 1517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:29:49.585821 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41915
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:49.933189 1529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:49.933497 1529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:49.933691 1529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:49.941282 1529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:49.941407 1529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:49.941598 1529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:49.951704 1529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:49.953172 1529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:49.954569 1529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:49.968063 1534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:49.968128 1537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:49.968091 1535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:49.968915 1529 server_base.cc:1034] running on GCE node
I20260925 01:29:49.969616 1529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:49.970669 1529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:49.971911 1529 hybrid_clock.cc:648] HybridClock initialized: now 1790299789971857 us; error 64 us; skew 500 ppm
I20260925 01:29:49.972251 1529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:49 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:49.975546 1529 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:29:49.977273 1529 webserver.cc:460] Webserver started at http://127.27.160.1:42721/ using document root <none> and password file <none>
I20260925 01:29:49.977967 1529 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:49.978111 1529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:49.978421 1529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:49.981455 1529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "8955e691d2af4ec3bccfc2a2c3345fa2"
format_stamp: "Formatted at 2026-09-25 01:29:49 on dist-test-slave-h6hf"
server_key: "d2f1126fd1798dda06a535127ebe5522"
server_key_iv: "7e621deb5d25ba0d67eec02a57f06b6f"
server_key_version: "encryptionkey@0"
I20260925 01:29:49.982300 1529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "8955e691d2af4ec3bccfc2a2c3345fa2"
format_stamp: "Formatted at 2026-09-25 01:29:49 on dist-test-slave-h6hf"
server_key: "d2f1126fd1798dda06a535127ebe5522"
server_key_iv: "7e621deb5d25ba0d67eec02a57f06b6f"
server_key_version: "encryptionkey@0"
I20260925 01:29:49.987790 1529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:29:49.991420 1544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:49.992733 1529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:29:49.992916 1529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "8955e691d2af4ec3bccfc2a2c3345fa2"
format_stamp: "Formatted at 2026-09-25 01:29:49 on dist-test-slave-h6hf"
server_key: "d2f1126fd1798dda06a535127ebe5522"
server_key_iv: "7e621deb5d25ba0d67eec02a57f06b6f"
server_key_version: "encryptionkey@0"
I20260925 01:29:49.993163 1529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:50.011174 1529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:50.016032 1529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:50.016376 1529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:50.017764 1529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:50.019434 1529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:50.019604 1529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:50.019800 1529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:50.019903 1529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:50.057363 1529 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:46381
I20260925 01:29:50.057420 1656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:46381 every 8 connection(s)
I20260925 01:29:50.058678 1529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:50.068149 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1529
I20260925 01:29:50.068358 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:50.069219 28288 external_mini_cluster.cc:1428] Setting key f8db3845fb53a7f02c8f1f3854947f08
Sep 25 01:29:50 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299789, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:50.082775 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:50.061346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:46235 (local address 127.27.160.62:41915)
0925 01:29:50.061795 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:50.061801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:50.063227 (+ 1426us) server_negotiation.cc:407] Connection header received
0925 01:29:50.065993 (+ 2766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:50.065998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:50.066090 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:50.066190 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:50.067741 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:50.068576 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:50.070804 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:50.071329 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:50.071463 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:50.075332 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:50.075338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:50.075341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:50.075375 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:50.078612 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:50.079623 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:50.079628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:50.079630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:50.079759 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:50.080312 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:50.080320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:50.080326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:50.080743 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:50.080902 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:50.081960 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0925 01:29:50.082266 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":200,"threads_started":1}
I20260925 01:29:50.083539 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:50.062014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41915 (local address 127.27.160.1:46235)
0925 01:29:50.063016 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:50.063068 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:50.065714 (+ 2646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:50.066402 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:50.066410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:50.066955 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:50.067560 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:50.067578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:50.068758 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:50.068765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:50.070661 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:50.070678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:50.071597 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:50.071606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:50.071841 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:50.072576 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:50.072604 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:50.075087 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:50.078765 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:50.078778 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:50.078802 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:50.079460 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:50.079874 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:50.079881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:50.079888 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:50.080144 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:50.080970 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:50.080976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:50.081522 (+ 546us) client_negotiation.cc:769] Sending connection context
0925 01:29:50.081805 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 01:29:50.082294 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":335,"threads_started":1}
I20260925 01:29:50.085148 1657 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41915
I20260925 01:29:50.085498 1657 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:50.086472 1657 heartbeater.cc:511] Master 127.27.160.62:41915 requested a full tablet report, sending...
I20260925 01:29:50.088866 1468 ts_manager.cc:194] Registered new tserver with Master: 8955e691d2af4ec3bccfc2a2c3345fa2 (127.27.160.1:46381)
I20260925 01:29:50.090353 1468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:46235
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:29:50.120836 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41915
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:50.454440 1663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:50.454783 1663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:50.454972 1663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:50.462359 1663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:50.462540 1663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:50.462724 1663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:50.472767 1663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:50.474226 1663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:50.475589 1663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:50.489459 1668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:50.489478 1669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:50.489598 1671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:50.490255 1663 server_base.cc:1034] running on GCE node
I20260925 01:29:50.490880 1663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:50.491861 1663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:50.493067 1663 hybrid_clock.cc:648] HybridClock initialized: now 1790299790493000 us; error 56 us; skew 500 ppm
I20260925 01:29:50.493438 1663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:50 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299790, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:50.497032 1663 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:29:50.498752 1663 webserver.cc:460] Webserver started at http://127.27.160.2:40819/ using document root <none> and password file <none>
I20260925 01:29:50.499472 1663 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:50.499606 1663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:50.499936 1663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:50.502987 1663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "ea98ed58a50e4a4ead3b14ed37865a4d"
format_stamp: "Formatted at 2026-09-25 01:29:50 on dist-test-slave-h6hf"
server_key: "6653a4185efa5b55c1961e3192d318d4"
server_key_iv: "b074d4b0a97d8b3476af838d3a17d69a"
server_key_version: "encryptionkey@0"
I20260925 01:29:50.503810 1663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "ea98ed58a50e4a4ead3b14ed37865a4d"
format_stamp: "Formatted at 2026-09-25 01:29:50 on dist-test-slave-h6hf"
server_key: "6653a4185efa5b55c1961e3192d318d4"
server_key_iv: "b074d4b0a97d8b3476af838d3a17d69a"
server_key_version: "encryptionkey@0"
I20260925 01:29:50.509306 1663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:29:50.513105 1678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:50.515058 1663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:29:50.515295 1663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "ea98ed58a50e4a4ead3b14ed37865a4d"
format_stamp: "Formatted at 2026-09-25 01:29:50 on dist-test-slave-h6hf"
server_key: "6653a4185efa5b55c1961e3192d318d4"
server_key_iv: "b074d4b0a97d8b3476af838d3a17d69a"
server_key_version: "encryptionkey@0"
I20260925 01:29:50.515573 1663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:50.538652 1663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:50.544269 1663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:50.544668 1663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:50.545940 1663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:50.547621 1663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:50.547713 1663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:50.547935 1663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:50.548002 1663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:50.587690 1663 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:41341
I20260925 01:29:50.587750 1790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:41341 every 8 connection(s)
I20260925 01:29:50.589279 1663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:50.592311 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1663
I20260925 01:29:50.592476 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:50.593156 28288 external_mini_cluster.cc:1428] Setting key 4c798e3274d0717febbc341bb8f932fe
Sep 25 01:29:50 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299790, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:29:50.613035 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:50.591995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:54271 (local address 127.27.160.62:41915)
0925 01:29:50.592573 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:50.592582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:50.593768 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 01:29:50.596219 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:50.596226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:50.596330 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:50.596468 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:50.598563 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:50.599439 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:50.601289 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:50.601793 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:50.601994 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:50.606200 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:50.606206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:50.606210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:50.606251 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:50.609315 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:50.610209 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:50.610219 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:50.610229 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:50.610377 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:50.610918 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:50.610930 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:50.610933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:50.611259 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:50.611459 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:50.612273 (+ 814us) server_negotiation.cc:299] Negotiation successful
0925 01:29:50.612535 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":318,"threads_started":1}
I20260925 01:29:50.614375 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:50.592627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41915 (local address 127.27.160.2:54271)
0925 01:29:50.593565 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:50.593622 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:50.595899 (+ 2277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:50.596784 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:50.596799 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:50.597580 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:50.598360 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:50.598386 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:50.599647 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:50.599653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:50.601083 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:50.601097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:50.602118 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:50.602128 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:50.602413 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:50.603230 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:50.603259 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:50.605893 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:50.609497 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:50.609508 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:50.609529 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:50.610052 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:50.610513 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:50.610520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:50.610524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:50.610766 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:50.611420 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:50.611430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:50.611843 (+ 413us) client_negotiation.cc:769] Sending connection context
0925 01:29:50.612169 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 01:29:50.612808 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":361,"threads_started":1}
I20260925 01:29:50.616118 1791 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
I20260925 01:29:50.616479 1791 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:50.617337 1791 heartbeater.cc:511] Master 127.27.160.62:41915 requested a full tablet report, sending...
I20260925 01:29:50.619024 1468 ts_manager.cc:194] Registered new tserver with Master: ea98ed58a50e4a4ead3b14ed37865a4d (127.27.160.2:41341)
I20260925 01:29:50.620339 1468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:54271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:29:50.644685 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41915
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:50.968538 1797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:50.968811 1797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:50.968966 1797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:50.976220 1797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:50.976330 1797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:50.976505 1797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:50.986862 1797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:50.988386 1797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:50.989831 1797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:51.002911 1802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:51.002842 1805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:51.002885 1803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:51.003711 1797 server_base.cc:1034] running on GCE node
I20260925 01:29:51.004310 1797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:51.005337 1797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:51.006592 1797 hybrid_clock.cc:648] HybridClock initialized: now 1790299791006508 us; error 54 us; skew 500 ppm
I20260925 01:29:51.006943 1797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:51.013738 1797 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:29:51.015317 1797 webserver.cc:460] Webserver started at http://127.27.160.3:42373/ using document root <none> and password file <none>
I20260925 01:29:51.016047 1797 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:51.016181 1797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:51.016517 1797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:51.019586 1797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "b4075cdda9f2421287d8e3ef659e05c6"
format_stamp: "Formatted at 2026-09-25 01:29:51 on dist-test-slave-h6hf"
server_key: "a3ce538ac91693973b205e81ee76b05c"
server_key_iv: "cdfc31c72783237494b60160e3525820"
server_key_version: "encryptionkey@0"
I20260925 01:29:51.020458 1797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "b4075cdda9f2421287d8e3ef659e05c6"
format_stamp: "Formatted at 2026-09-25 01:29:51 on dist-test-slave-h6hf"
server_key: "a3ce538ac91693973b205e81ee76b05c"
server_key_iv: "cdfc31c72783237494b60160e3525820"
server_key_version: "encryptionkey@0"
I20260925 01:29:51.026044 1797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 01:29:51.029625 1812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:51.031090 1797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:29:51.031255 1797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "b4075cdda9f2421287d8e3ef659e05c6"
format_stamp: "Formatted at 2026-09-25 01:29:51 on dist-test-slave-h6hf"
server_key: "a3ce538ac91693973b205e81ee76b05c"
server_key_iv: "cdfc31c72783237494b60160e3525820"
server_key_version: "encryptionkey@0"
I20260925 01:29:51.031461 1797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:51.055256 1797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:51.059813 1797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:51.060139 1797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:51.061281 1797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:51.062578 1797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:51.062681 1797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:51.062829 1797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:51.062901 1797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:51.093575 1657 heartbeater.cc:503] Master 127.27.160.62:41915 was elected leader, sending a full tablet report...
I20260925 01:29:51.096473 1797 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:38715
I20260925 01:29:51.096519 1924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:38715 every 8 connection(s)
I20260925 01:29:51.098017 1797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:51.104952 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1797
I20260925 01:29:51.105176 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:51.105764 28288 external_mini_cluster.cc:1428] Setting key 89e479a0e33cb9bd110a74abc45c9a76
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:29:51.121606 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:51.100504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:47543 (local address 127.27.160.62:41915)
0925 01:29:51.100716 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.100722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.102079 (+ 1357us) server_negotiation.cc:407] Connection header received
0925 01:29:51.103823 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.103827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.103895 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.103982 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.106009 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.106927 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.108945 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.109402 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.109501 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:51.114004 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:51.114011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:51.114024 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:51.114064 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:51.117636 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.118579 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.118598 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.118602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.118760 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.119300 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.119305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.119309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.119698 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:51.119857 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:51.120959 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0925 01:29:51.121237 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:29:51.122529 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:51.101172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41915 (local address 127.27.160.3:47543)
0925 01:29:51.101911 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:51.101949 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:51.103577 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:51.104201 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:51.104209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:51.104924 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:51.105817 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:51.105846 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.107105 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.107109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:51.108810 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:51.108821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.109589 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.109594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:51.109890 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:51.110722 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:51.110756 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:51.113744 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:51.117801 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:51.117810 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:51.117829 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:51.118388 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:51.118882 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:51.118889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:51.118894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:51.119150 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:51.119928 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:51.119936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:51.120474 (+ 538us) client_negotiation.cc:769] Sending connection context
0925 01:29:51.120822 (+ 348us) client_negotiation.cc:240] Negotiation successful
0925 01:29:51.121348 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":288,"threads_started":1}
I20260925 01:29:51.123811 1925 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41915
I20260925 01:29:51.124096 1925 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:51.124751 1925 heartbeater.cc:511] Master 127.27.160.62:41915 requested a full tablet report, sending...
I20260925 01:29:51.126127 1468 ts_manager.cc:194] Registered new tserver with Master: b4075cdda9f2421287d8e3ef659e05c6 (127.27.160.3:38715)
I20260925 01:29:51.126873 1468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:47543
I20260925 01:29:51.134795 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/test-admin.keytab.
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:51.162508 28288 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 01:29:51.167812 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:29:51.165331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.27.160.1:46381)
0925 01:29:51.165983 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.165990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.166017 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:29:51.166084 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.166091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.166305 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.166434 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.167403 (+ 969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36554: BlockingRecv error: recv got EOF from 127.0.0.1:36554 (error 108)
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":336,"threads_started":1}
W20260925 01:29:51.167804 28288 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5cc' of type FILE
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:51.169448 28288 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:51.170786 28288 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:51.186095 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:51.172460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.27.160.1:46381)
0925 01:29:51.172654 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.172659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.172800 (+ 141us) server_negotiation.cc:407] Connection header received
0925 01:29:51.172930 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.172934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.172991 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.173086 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.174160 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.175190 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.176377 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.176922 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.177020 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:51.180618 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:51.180636 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:51.180656 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:51.180694 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:51.183641 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.184244 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.184251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.184254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.184369 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.184682 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.184685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.184687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.185022 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:51.185307 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:51.185538 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 01:29:51.185788 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 01:29:51.191821 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:29:51.190243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.27.160.1:46381)
0925 01:29:51.190463 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.190469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.190682 (+ 213us) server_negotiation.cc:407] Connection header received
0925 01:29:51.190794 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.190797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.190863 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.190939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.191573 (+ 634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36564: BlockingRecv error: recv got EOF from 127.0.0.1:36564 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 01:29:51.191923 28288 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5cc' of type FILE
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:51.193496 28288 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:51.194756 28288 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:51.208266 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:51.195919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.27.160.1:46381)
0925 01:29:51.196109 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.196114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.196290 (+ 176us) server_negotiation.cc:407] Connection header received
0925 01:29:51.196451 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.196454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.196506 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.196593 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.197701 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.198609 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.199760 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.200186 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.200278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:51.203632 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:51.203637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:51.203640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:51.203670 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:51.206142 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.206717 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.206721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.206723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.206834 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.207135 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.207138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.207140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.207380 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:51.207571 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:51.207790 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 01:29:51.207967 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 01:29:51.212523 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:29:51.210845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.27.160.1:46381)
0925 01:29:51.211036 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.211042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.211236 (+ 194us) server_negotiation.cc:407] Connection header received
0925 01:29:51.211412 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.211423 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.211487 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.211592 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.212292 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36580: BlockingRecv error: recv got EOF from 127.0.0.1:36580 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 01:29:51.213018 28288 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790299737515179-28288-0/krb5kdc/krb5cc)
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:51.214480 28288 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:51.215739 28288 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:29:51.229604 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:51.216905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.27.160.1:46381)
0925 01:29:51.217163 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.217169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.217355 (+ 186us) server_negotiation.cc:407] Connection header received
0925 01:29:51.217461 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.217465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.217512 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.217588 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.218399 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.219087 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.220469 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.220846 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.220932 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:51.224370 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:51.224375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:51.224378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:51.224411 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:51.227149 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.227740 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.227744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.227746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.227961 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.228271 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.228274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.228276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.228569 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:51.228805 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:51.229016 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 01:29:51.229266 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:29:51.231273 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1529
I20260925 01:29:51.242532 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:51.231909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.27.160.1:46381)
0925 01:29:51.232066 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.232070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.232429 (+ 359us) server_negotiation.cc:407] Connection header received
0925 01:29:51.232549 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.232552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.232602 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.232676 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.233663 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.234316 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.235515 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.235921 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.236020 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:51.238089 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:51.238094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:51.238097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:51.238124 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:51.240355 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.240915 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.240918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.240921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.241060 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.241429 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.241432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.241434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.241671 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:51.241856 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:51.242125 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 01:29:51.242279 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:29:51.243101 1652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:51.325786 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1529
I20260925 01:29:51.342931 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1663
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:29:51.358083 1945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:51.343748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.27.160.2:41341)
0925 01:29:51.344314 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.344322 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.344348 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:29:51.344411 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.344418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.344645 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.344777 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.345796 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.346788 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.347973 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.348503 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.348604 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:51.352351 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:51.352363 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:51.352379 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:51.352410 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:51.355472 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.356111 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.356118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.356120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.356236 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.356553 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.356556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.356557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.356887 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:51.357173 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:51.357493 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 01:29:51.357764 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":314,"threads_started":1}
I20260925 01:29:51.359367 1786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:51.442528 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1663
I20260925 01:29:51.460763 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1797
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:29:51.476348 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:51.461980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.27.160.3:38715)
0925 01:29:51.462721 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:51.462728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:51.462759 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:29:51.462819 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:51.462825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:51.463051 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:51.463180 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:51.464189 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.465319 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.466550 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:51.467114 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:51.467214 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:51.471148 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:51.471160 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:51.471175 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:51.471215 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:51.474005 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.474634 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.474641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.474643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.474759 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:51.475069 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:51.475073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:51.475075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:51.475397 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:51.475633 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:51.475867 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 01:29:51.476071 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":435,"threads_started":1}
I20260925 01:29:51.477717 1920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:51.562903 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1797
I20260925 01:29:51.578998 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1436
I20260925 01:29:51.580411 1498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:51.686124 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1436
2026-09-25T01:29:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4022 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1952](info): set up 2 sockets
Sep 25 01:29:51 dist-test-slave-h6hf krb5kdc[1952](info): commencing operation
krb5kdc: starting...
W20260925 01:29:52.803038 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:29:52 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:29:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:29:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.62.keytab.
Entry for principal oryx/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.62.keytab.
I20260925 01:29:53.021392 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:33983
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.62.keytab
--principal=oryx/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:33983
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:53.350951 1965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:53.351315 1965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:53.351513 1965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:53.358848 1965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:53.359041 1965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:53.359145 1965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:53.359253 1965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:53.359345 1965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:53.369293 1965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:33983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.62.keytab
--principal=oryx/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:53.370711 1965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:53.372098 1965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:53.384284 1973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:53.384479 1970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:53.384423 1971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:53.385794 1965 server_base.cc:1034] running on GCE node
I20260925 01:29:53.386722 1965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:53.388284 1965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:53.389525 1965 hybrid_clock.cc:648] HybridClock initialized: now 1790299793389475 us; error 50 us; skew 500 ppm
I20260925 01:29:53.389886 1965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:53 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299793, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:53.393273 1965 init.cc:377] Logged in from keytab as oryx/127.27.160.62@KRBTEST.COM (short username oryx)
I20260925 01:29:53.394938 1965 webserver.cc:460] Webserver started at http://127.27.160.62:35505/ using document root <none> and password file <none>
I20260925 01:29:53.395684 1965 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:53.395843 1965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:53.396166 1965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:53.399310 1965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "5e268b00cbb64f1fa59b486526de6a63"
format_stamp: "Formatted at 2026-09-25 01:29:53 on dist-test-slave-h6hf"
server_key: "72d07696776f455a8522d971494f03ba"
server_key_iv: "1a517a80eaa095a1c1ac25c75d937bf8"
server_key_version: "encryptionkey@0"
I20260925 01:29:53.400141 1965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "5e268b00cbb64f1fa59b486526de6a63"
format_stamp: "Formatted at 2026-09-25 01:29:53 on dist-test-slave-h6hf"
server_key: "72d07696776f455a8522d971494f03ba"
server_key_iv: "1a517a80eaa095a1c1ac25c75d937bf8"
server_key_version: "encryptionkey@0"
I20260925 01:29:53.405987 1965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:29:53.409700 1980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:53.411100 1965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:29:53.411283 1965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "5e268b00cbb64f1fa59b486526de6a63"
format_stamp: "Formatted at 2026-09-25 01:29:53 on dist-test-slave-h6hf"
server_key: "72d07696776f455a8522d971494f03ba"
server_key_iv: "1a517a80eaa095a1c1ac25c75d937bf8"
server_key_version: "encryptionkey@0"
I20260925 01:29:53.411458 1965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:53.445322 1965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:53.450184 1965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:53.450459 1965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:53.470919 1965 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:33983
I20260925 01:29:53.470937 2031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:33983 every 8 connection(s)
I20260925 01:29:53.472544 1965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:29:53.476540 2032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:53.482990 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 1965
I20260925 01:29:53.483251 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:29:53.483922 28288 external_mini_cluster.cc:1428] Setting key 58fa5cbc5d456f70af08f35b63652990
I20260925 01:29:53.487352 2032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: Bootstrap starting.
I20260925 01:29:53.491283 2032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:53.492698 2032 log.cc:824] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:53.495591 2032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: No bootstrap required, opened a new log
Sep 25 01:29:53 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:29:53.503796 2032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } }
I20260925 01:29:53.504297 2032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:53.504415 2032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e268b00cbb64f1fa59b486526de6a63, State: Initialized, Role: FOLLOWER
I20260925 01:29:53.505105 2032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } }
I20260925 01:29:53.505412 2032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:29:53.505551 2032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:29:53.505744 2032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:53.508338 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:53.485969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.27.160.62:33983)
0925 01:29:53.486804 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:53.486822 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:53.486876 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:29:53.489294 (+ 2418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:53.489323 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:53.489846 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:53.490290 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:53.491746 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:53.493726 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:53.495026 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:53.495751 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:53.495954 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:53.499981 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:53.499989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:53.500008 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:53.500054 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:53.503326 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:53.504164 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:53.504170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:53.504203 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:53.504460 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:53.504919 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:53.504929 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:53.504932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:53.505701 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:53.506030 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:53.506427 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 01:29:53.506914 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":339,"threads_started":1}
I20260925 01:29:53.509174 2032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } }
I20260925 01:29:53.509757 2032 leader_election.cc:302] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e268b00cbb64f1fa59b486526de6a63; no voters:
I20260925 01:29:53.510502 2032 leader_election.cc:288] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:29:53.510645 2037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:53.511328 2037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 1 LEADER]: Becoming Leader. State: Replica: 5e268b00cbb64f1fa59b486526de6a63, State: Running, Role: LEADER
I20260925 01:29:53.511965 2037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } }
I20260925 01:29:53.514176 2032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:53.515103 2039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } }
I20260925 01:29:53.515445 2039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:53.517771 2038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e268b00cbb64f1fa59b486526de6a63. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } }
I20260925 01:29:53.518175 2038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:53.520975 2042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:29:53.526583 2042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:29:53.540989 2042 catalog_manager.cc:1347] Generated new cluster ID: f4ff7f0d1de44a2d928d3ed9de76f417
I20260925 01:29:53.541205 2042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:29:53.555318 2042 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:29:53.556697 2042 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:29:53.566102 2042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: Generated new TSK 0
I20260925 01:29:53.567509 2042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab.
Entry for principal oryx/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab.
I20260925 01:29:53.628417 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab
--principal=oryx/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:53.974354 2058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:53.974663 2058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:53.974884 2058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:53.982442 2058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:53.982563 2058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:53.982795 2058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:53.992833 2058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab
--principal=oryx/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:53.994359 2058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:53.995751 2058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:54.010121 2066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:54.010070 2064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:54.010568 2063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:54.011034 2058 server_base.cc:1034] running on GCE node
I20260925 01:29:54.011662 2058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:54.012620 2058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:54.013818 2058 hybrid_clock.cc:648] HybridClock initialized: now 1790299794013725 us; error 85 us; skew 500 ppm
I20260925 01:29:54.014209 2058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:54 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299794, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:54.017782 2058 init.cc:377] Logged in from keytab as oryx/127.27.160.1@KRBTEST.COM (short username oryx)
I20260925 01:29:54.019481 2058 webserver.cc:460] Webserver started at http://127.27.160.1:33737/ using document root <none> and password file <none>
I20260925 01:29:54.020157 2058 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:54.020260 2058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:54.020510 2058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:54.023623 2058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "764191dcd7ec42758b4181b222acd471"
format_stamp: "Formatted at 2026-09-25 01:29:54 on dist-test-slave-h6hf"
server_key: "b4f3a762307699fa861af098681ebf71"
server_key_iv: "b460a89fe7a24cadd6537f9c74746743"
server_key_version: "encryptionkey@0"
I20260925 01:29:54.024439 2058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "764191dcd7ec42758b4181b222acd471"
format_stamp: "Formatted at 2026-09-25 01:29:54 on dist-test-slave-h6hf"
server_key: "b4f3a762307699fa861af098681ebf71"
server_key_iv: "b460a89fe7a24cadd6537f9c74746743"
server_key_version: "encryptionkey@0"
I20260925 01:29:54.029927 2058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 01:29:54.033461 2073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:54.034829 2058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:29:54.035015 2058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "764191dcd7ec42758b4181b222acd471"
format_stamp: "Formatted at 2026-09-25 01:29:54 on dist-test-slave-h6hf"
server_key: "b4f3a762307699fa861af098681ebf71"
server_key_iv: "b460a89fe7a24cadd6537f9c74746743"
server_key_version: "encryptionkey@0"
I20260925 01:29:54.035239 2058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:54.051668 2058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:54.056131 2058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:54.056429 2058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:54.057508 2058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:54.058936 2058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:54.059036 2058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:54.059135 2058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:54.059190 2058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:54.094985 2058 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:40581
I20260925 01:29:54.095021 2185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:40581 every 8 connection(s)
I20260925 01:29:54.096426 2058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:54.100977 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 2058
I20260925 01:29:54.101230 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:29:54.101938 28288 external_mini_cluster.cc:1428] Setting key 9ed98d481a5cb3d0ac30dab24234955b
Sep 25 01:29:54 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299794, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:29:54.117901 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:54.099112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:51139 (local address 127.27.160.62:33983)
0925 01:29:54.099586 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:54.099590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:54.100521 (+ 931us) server_negotiation.cc:407] Connection header received
0925 01:29:54.102514 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:54.102520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:54.102612 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:54.102729 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:54.104179 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:54.105042 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:54.106604 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:54.106983 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:54.107132 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:54.110890 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:54.110899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:54.110907 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:54.110984 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:54.114148 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:54.115181 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:54.115188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:54.115192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:54.115368 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:54.115957 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:54.115968 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:54.115970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:54.116273 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:54.116449 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:54.117232 (+ 783us) server_negotiation.cc:299] Negotiation successful
0925 01:29:54.117480 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260925 01:29:54.118999 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:54.099628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.1:51139)
0925 01:29:54.100358 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:54.100401 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:54.102249 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:54.102964 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:54.102973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:54.103453 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:54.104008 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:54.104028 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:54.105226 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:54.105230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:54.106440 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:54.106452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:54.107217 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:54.107226 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:54.107467 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:54.108182 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:54.108205 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:54.110640 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:54.114396 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:54.114403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:54.114418 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:54.115041 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:54.115506 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:54.115513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:54.115519 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:54.115797 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:54.116422 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:54.116431 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:54.116795 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 01:29:54.117104 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 01:29:54.117680 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":291,"threads_started":1}
I20260925 01:29:54.120777 2186 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:29:54.121184 2186 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:54.122172 2186 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:29:54.124634 1997 ts_manager.cc:194] Registered new tserver with Master: 764191dcd7ec42758b4181b222acd471 (127.27.160.1:40581)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.2@KRBTEST.COM" created.
I20260925 01:29:54.126312 1997 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:51139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab.
Entry for principal oryx/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab.
I20260925 01:29:54.152756 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab
--principal=oryx/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:54.476221 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:54.476504 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:54.476666 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:54.483958 2192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:54.484068 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:54.484252 2192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:54.494128 2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab
--principal=oryx/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:54.495469 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:54.496858 2192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:54.510254 2197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:54.510177 2200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:54.510231 2198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:54.511034 2192 server_base.cc:1034] running on GCE node
I20260925 01:29:54.511641 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:54.512571 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:54.513784 2192 hybrid_clock.cc:648] HybridClock initialized: now 1790299794513729 us; error 50 us; skew 500 ppm
I20260925 01:29:54.514087 2192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:54 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299794, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:54.517683 2192 init.cc:377] Logged in from keytab as oryx/127.27.160.2@KRBTEST.COM (short username oryx)
I20260925 01:29:54.519352 2192 webserver.cc:460] Webserver started at http://127.27.160.2:43655/ using document root <none> and password file <none>
I20260925 01:29:54.520079 2192 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:54.520217 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:54.520533 2192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:54.523780 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "22c8f4062454451b8404bb1cf3f389fc"
format_stamp: "Formatted at 2026-09-25 01:29:54 on dist-test-slave-h6hf"
server_key: "5843ae7e2a684a41220c4fcf75c1e3ae"
server_key_iv: "f0b83e84371f8c692cc726e90cb410c6"
server_key_version: "encryptionkey@0"
I20260925 01:29:54.524569 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "22c8f4062454451b8404bb1cf3f389fc"
format_stamp: "Formatted at 2026-09-25 01:29:54 on dist-test-slave-h6hf"
server_key: "5843ae7e2a684a41220c4fcf75c1e3ae"
server_key_iv: "f0b83e84371f8c692cc726e90cb410c6"
server_key_version: "encryptionkey@0"
I20260925 01:29:54.530298 2192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 01:29:54.535964 2207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:54.538374 2192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 01:29:54.538590 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "22c8f4062454451b8404bb1cf3f389fc"
format_stamp: "Formatted at 2026-09-25 01:29:54 on dist-test-slave-h6hf"
server_key: "5843ae7e2a684a41220c4fcf75c1e3ae"
server_key_iv: "f0b83e84371f8c692cc726e90cb410c6"
server_key_version: "encryptionkey@0"
I20260925 01:29:54.538870 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:54.566780 2192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:54.572053 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:54.572408 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:54.573571 2192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:54.575243 2192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:54.575326 2192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:54.575575 2192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:54.575636 2192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:54.611944 2192 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:36355
I20260925 01:29:54.611980 2319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:36355 every 8 connection(s)
I20260925 01:29:54.613427 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:54.623167 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 2192
I20260925 01:29:54.623369 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:29:54.623802 28288 external_mini_cluster.cc:1428] Setting key 726984540042606b082665e55febc984
Sep 25 01:29:54 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299794, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:29:54.634585 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:54.616157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:60821 (local address 127.27.160.62:33983)
0925 01:29:54.616331 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:54.616335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:54.617628 (+ 1293us) server_negotiation.cc:407] Connection header received
0925 01:29:54.619468 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:54.619474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:54.619577 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:54.619688 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:54.621490 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:54.622212 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:54.623871 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:54.624332 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:54.624475 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:54.628251 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:54.628256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:54.628260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:54.628293 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:54.630947 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:54.631786 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:54.631794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:54.631803 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:54.631930 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:54.632396 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:54.632405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:54.632407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:54.632819 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:54.632984 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:54.633856 (+ 872us) server_negotiation.cc:299] Negotiation successful
0925 01:29:54.634134 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:29:54.635293 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:54.616697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.2:60821)
0925 01:29:54.617457 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:54.617500 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:54.619190 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:54.619960 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:54.619972 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:54.620628 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:54.621311 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:54.621332 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:54.622370 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:54.622380 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:54.623727 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:54.623740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:54.624568 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:54.624575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:54.624804 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:54.625520 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:54.625547 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:54.628004 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:54.631155 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:54.631165 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:54.631192 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:54.631645 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:54.632048 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:54.632053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:54.632056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:54.632266 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:54.632971 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:54.632978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:54.633414 (+ 436us) client_negotiation.cc:769] Sending connection context
0925 01:29:54.633695 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 01:29:54.634127 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":281,"threads_started":1}
I20260925 01:29:54.636843 2320 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:29:54.637205 2320 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:54.638129 2320 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
I20260925 01:29:54.639786 1997 ts_manager.cc:194] Registered new tserver with Master: 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2:36355)
I20260925 01:29:54.640642 1997 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:60821
WARNING: no policy specified for oryx/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab.
Entry for principal oryx/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab.
I20260925 01:29:54.676476 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab
--principal=oryx/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:55.004655 2325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:55.005060 2325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:55.005288 2325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:55.012827 2325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:55.012998 2325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:55.013252 2325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:55.023798 2325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab
--principal=oryx/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:55.025238 2325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:55.026592 2325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:55.041241 2331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:55.041324 2333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:55.041915 2330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:55.042862 2325 server_base.cc:1034] running on GCE node
I20260925 01:29:55.043588 2325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:55.044752 2325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:55.046023 2325 hybrid_clock.cc:648] HybridClock initialized: now 1790299795045958 us; error 50 us; skew 500 ppm
I20260925 01:29:55.046394 2325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:55 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299795, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:55.049947 2325 init.cc:377] Logged in from keytab as oryx/127.27.160.3@KRBTEST.COM (short username oryx)
I20260925 01:29:55.051573 2325 webserver.cc:460] Webserver started at http://127.27.160.3:45097/ using document root <none> and password file <none>
I20260925 01:29:55.052358 2325 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:55.052529 2325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:55.052883 2325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:55.056293 2325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "5b805c41709d45b1a9cd4d074a450adf"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "982924fc466b89a07bc14d402440aaa6"
server_key_iv: "381876c70c7d013c096566b075f01818"
server_key_version: "encryptionkey@0"
I20260925 01:29:55.057232 2325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "5b805c41709d45b1a9cd4d074a450adf"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "982924fc466b89a07bc14d402440aaa6"
server_key_iv: "381876c70c7d013c096566b075f01818"
server_key_version: "encryptionkey@0"
I20260925 01:29:55.062916 2325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:29:55.066838 2340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:55.068315 2325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:29:55.068540 2325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "5b805c41709d45b1a9cd4d074a450adf"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "982924fc466b89a07bc14d402440aaa6"
server_key_iv: "381876c70c7d013c096566b075f01818"
server_key_version: "encryptionkey@0"
I20260925 01:29:55.068779 2325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:55.093719 2325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:55.098865 2325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:55.099207 2325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:55.100245 2325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:55.101694 2325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:55.101810 2325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:55.101997 2325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:55.102094 2325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:55.131835 2186 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
I20260925 01:29:55.139977 2325 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:33863
I20260925 01:29:55.140033 2452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:33863 every 8 connection(s)
I20260925 01:29:55.141337 2325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:55.147591 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 2325
I20260925 01:29:55.147787 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:29:55.148253 28288 external_mini_cluster.cc:1428] Setting key b2030ed66c41a38a51eb676a0e6a808c
Sep 25 01:29:55 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299795, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:29:55.161223 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.143893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:60459 (local address 127.27.160.62:33983)
0925 01:29:55.144318 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:55.144325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:55.145479 (+ 1154us) server_negotiation.cc:407] Connection header received
0925 01:29:55.147250 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:55.147256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:55.147349 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:55.147464 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:55.149161 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.150276 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.151932 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.152321 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.152420 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.155779 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:55.155785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:55.155787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:55.155820 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:55.158588 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.159233 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.159237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.159240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.159348 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.159725 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.159728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.159730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.159969 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:55.160104 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:55.160724 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 01:29:55.160876 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":186,"threads_started":1}
I20260925 01:29:55.162400 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.144518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.3:60459)
0925 01:29:55.145318 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:55.145356 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:55.146991 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:55.147703 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:55.147736 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:55.148293 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:55.148915 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.148949 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.150427 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.150431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.151754 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.151766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.152507 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.152512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.152718 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.153419 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:55.153440 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:55.155589 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:55.158708 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.158714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.158727 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.159133 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.159440 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.159443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.159445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.159634 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.160096 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:55.160101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:55.160397 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 01:29:55.160637 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 01:29:55.161006 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":286,"threads_started":1}
I20260925 01:29:55.163839 2453 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:29:55.164145 2453 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:55.164829 2453 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
I20260925 01:29:55.166245 1996 ts_manager.cc:194] Registered new tserver with Master: 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863)
I20260925 01:29:55.167140 1996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:60459
I20260925 01:29:55.175782 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.160.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.61.keytab.
Entry for principal oryx/127.27.160.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.61.keytab.
I20260925 01:29:55.224610 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.61:37869
--webserver_interface=127.27.160.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.61.keytab
--principal=oryx/127.27.160.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:33983,127.27.160.61:37869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:29:55.563131 2459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:55.563441 2459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:55.563616 2459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:55.571174 2459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:55.571298 2459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:55.571398 2459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:55.571494 2459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:55.571570 2459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:55.581990 2459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:33983,127.27.160.61:37869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.61.keytab
--principal=oryx/127.27.160.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.61:37869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.160.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
W20260925 01:29:55.582407 2459 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.160.62:33983,127.27.160.61:37869'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 01:29:55.583498 2459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:55.584895 2459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:55.597096 2464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:55.597154 2467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:55.597177 2465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:55.598090 2459 server_base.cc:1034] running on GCE node
I20260925 01:29:55.598979 2459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:55.600507 2459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:55.601745 2459 hybrid_clock.cc:648] HybridClock initialized: now 1790299795601666 us; error 59 us; skew 500 ppm
I20260925 01:29:55.602084 2459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:55 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299795, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:55.605455 2459 init.cc:377] Logged in from keytab as oryx/127.27.160.61@KRBTEST.COM (short username oryx)
I20260925 01:29:55.606985 2459 webserver.cc:460] Webserver started at http://127.27.160.61:38761/ using document root <none> and password file <none>
I20260925 01:29:55.607728 2459 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:55.607856 2459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:55.608265 2459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:55.611299 2459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data/instance:
uuid: "4017e314eba74525b086cc1dcbd5e905"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "90808429901203228ac358f6cae3cd74"
server_key_iv: "f13ee50345052f62fca0d4fa98a547f0"
server_key_version: "encryptionkey@0"
I20260925 01:29:55.612180 2459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal/instance:
uuid: "4017e314eba74525b086cc1dcbd5e905"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "90808429901203228ac358f6cae3cd74"
server_key_iv: "f13ee50345052f62fca0d4fa98a547f0"
server_key_version: "encryptionkey@0"
I20260925 01:29:55.617729 2459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:29:55.621392 2474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:55.622965 2459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:29:55.623172 2459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
uuid: "4017e314eba74525b086cc1dcbd5e905"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "90808429901203228ac358f6cae3cd74"
server_key_iv: "f13ee50345052f62fca0d4fa98a547f0"
server_key_version: "encryptionkey@0"
I20260925 01:29:55.623416 2459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:55.644204 2320 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
I20260925 01:29:55.646270 2459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:55.650792 2459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:55.651083 2459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:55.670548 2459 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.61:37869
I20260925 01:29:55.670578 2525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.61:37869 every 8 connection(s)
I20260925 01:29:55.672009 2459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
I20260925 01:29:55.675351 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 2459
I20260925 01:29:55.676610 2526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:55.686719 2526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:29:55 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.61@KRBTEST.COM
Sep 25 01:29:55 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299795, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.61@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:29:55.700356 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.677620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.27.160.61:37869)
0925 01:29:55.678434 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:55.678450 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:55.678505 (+ 55us) server_negotiation.cc:407] Connection header received
0925 01:29:55.680279 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:55.680307 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:55.680866 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:55.681380 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:55.682704 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.684772 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.686103 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.687165 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.687365 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.692341 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:55.692353 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:55.692364 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:55.692400 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:55.695851 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.696709 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.696715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.696721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.696893 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.697342 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.697347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.697350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.698061 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:55.698401 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:55.698711 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:29:55.699115 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":329,"threads_started":1}
I20260925 01:29:55.705503 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.689717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.27.160.62:33983)
0925 01:29:55.690232 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:55.690239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:55.690809 (+ 570us) server_negotiation.cc:407] Connection header received
0925 01:29:55.691020 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:55.691023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:55.691107 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:55.691223 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:55.692710 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.693882 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.695428 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.696091 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.696182 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.699995 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:55.700002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:55.700007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:55.700052 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:55.703008 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.703773 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.703778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.703780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.703898 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.704207 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.704210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.704212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.704511 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:55.704633 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:55.705025 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 01:29:55.705261 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":267,"threads_started":1}
I20260925 01:29:55.705523 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.689791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:56880)
0925 01:29:55.690644 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:55.690693 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:55.690876 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:55.691323 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:55.691333 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:55.692226 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:55.692556 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.692569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.694021 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.694027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.695282 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.695295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.696272 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.696279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.696471 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.697218 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:55.697235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:55.699807 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:55.703156 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.703167 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.703174 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.703664 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.703991 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.703994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.703996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.704120 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.704614 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:55.704619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:55.704762 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 01:29:55.704937 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 01:29:55.705233 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":434,"threads_started":1}
I20260925 01:29:55.707103 2526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:29:55 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299795, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.61@KRBTEST.COM for oryx/127.27.160.61@KRBTEST.COM
I20260925 01:29:55.718752 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.707810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.0.0.1:43444)
0925 01:29:55.707980 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:55.708003 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:55.708088 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:55.708493 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:55.708496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:55.708805 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:55.708919 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.708927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.709903 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.709907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.710774 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.710785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.711386 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.711390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.711512 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.712009 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:55.712024 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:55.713972 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:55.716751 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.716754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.716757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.717167 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.717495 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.717498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.717500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.717625 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.718059 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:55.718062 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:55.718182 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 01:29:55.718287 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:29:55.718498 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 01:29:55.718992 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.707750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.27.160.61:37869)
0925 01:29:55.707900 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:55.707904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:55.708059 (+ 155us) server_negotiation.cc:407] Connection header received
0925 01:29:55.708169 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:55.708172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:55.708279 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:55.708375 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:55.709042 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.709784 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.710886 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.711271 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.711360 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.714160 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:55.714165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:55.714168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:55.714196 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:55.716625 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.717276 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.717279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.717282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.717408 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.717708 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.717711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.717713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.717969 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:55.718080 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:55.718363 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 01:29:55.718620 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":54}
I20260925 01:29:55.723707 2526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Bootstrap starting.
I20260925 01:29:55.726948 2526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:55.728092 2526 log.cc:824] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:55.730775 2526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: No bootstrap required, opened a new log
I20260925 01:29:55.737831 2526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } }
I20260925 01:29:55.738173 2526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:55.738270 2526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4017e314eba74525b086cc1dcbd5e905, State: Initialized, Role: FOLLOWER
I20260925 01:29:55.738855 2526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } }
I20260925 01:29:55.739799 2533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } } }
I20260925 01:29:55.740192 2533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:29:55.740882 2526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:29:55.749599 2544 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:29:55.749758 2544 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:29:55.752175 2459 master_runner.cc:428] Detected that this master 4017e314eba74525b086cc1dcbd5e905 is joining an existing cluster
I20260925 01:29:55.752285 2459 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.160.61:37869
WARNING: no policy specified for oryx/127.27.160.60@KRBTEST.COM; defaulting to no policy
I20260925 01:29:55.782676 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.757418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.27.160.61:37869)
0925 01:29:55.757591 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:55.757598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:55.766557 (+ 8959us) server_negotiation.cc:407] Connection header received
0925 01:29:55.766759 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:55.766766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:55.766850 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:55.766921 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:55.768406 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.769602 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.771782 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.772669 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.772795 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.775479 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:55.775488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:55.775495 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:55.775535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:55.779527 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.780475 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.780480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.780484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.780667 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.781102 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.781107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.781110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.781533 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:55.781696 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:55.782226 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 01:29:55.782414 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":32}
I20260925 01:29:55.782685 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.757724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.27.160.62:33983)
0925 01:29:55.757918 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:55.757927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:55.766025 (+ 8098us) server_negotiation.cc:407] Connection header received
0925 01:29:55.766241 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:55.766248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:55.766352 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:55.766462 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:55.767920 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.769045 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.771219 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.771895 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.772004 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.775137 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:55.775144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:55.775149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:55.775193 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:55.779241 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.780193 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.780199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.780203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.780363 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.780839 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.780844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.780847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.781224 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:55.781362 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:55.782057 (+ 695us) server_negotiation.cc:299] Negotiation successful
0925 01:29:55.782299 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 01:29:55.783779 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.756962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.0.0.1:43456)
0925 01:29:55.766472 (+ 9510us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:55.766496 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:55.766638 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:55.767132 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:55.767139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:55.767897 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:55.768254 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.768269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.769822 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.769830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.771635 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.771652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.773073 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.773081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.773305 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.773982 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:55.774009 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:55.775290 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:55.779685 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.779689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.779693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.780343 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.780804 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.780808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.780815 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.780978 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.781775 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:55.781781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:55.781947 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 01:29:55.782977 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0925 01:29:55.783449 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9370,"mutex_wait_us":72,"spinlock_wait_cycles":26368,"thread_start_us":8342,"threads_started":1}
I20260925 01:29:55.784184 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.756945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:56892)
0925 01:29:55.765891 (+ 8946us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:55.765908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:55.766101 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:55.766540 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:55.766546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:55.767280 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:55.767723 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.767738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.769287 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.769292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.771046 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.771061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.772317 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.772323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.772598 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.773348 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:55.773371 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:55.774910 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:55.779436 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.779441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.779446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.780044 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.780509 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.780525 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.780529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.780708 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.781440 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:55.781445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:55.781676 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 01:29:55.783232 (+ 1556us) client_negotiation.cc:240] Negotiation successful
0925 01:29:55.783683 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8810,"mutex_wait_us":86,"thread_start_us":8602,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.60@KRBTEST.COM" created.
Sep 25 01:29:55 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299793, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.62@KRBTEST.COM for oryx/127.27.160.61@KRBTEST.COM
I20260925 01:29:55.812795 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.791812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.0.0.1:43458)
0925 01:29:55.792563 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:55.792617 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:55.792814 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:55.793421 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:55.793433 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:55.794486 (+ 1053us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:55.794876 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.794892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.796718 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.796727 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.798908 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.798928 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.800026 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.800036 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.800316 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.801165 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:55.801197 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:55.804766 (+ 3569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:55.809549 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.809569 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.809578 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.810321 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.810785 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.810790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.810796 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.810993 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.811663 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:55.811670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:55.811840 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 01:29:55.812030 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 01:29:55.812409 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":314,"threads_started":1}
I20260925 01:29:55.812875 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.791860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.27.160.61:37869)
0925 01:29:55.792058 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:55.792064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:55.792721 (+ 657us) server_negotiation.cc:407] Connection header received
0925 01:29:55.793076 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:55.793087 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:55.793229 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:55.793339 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:55.795149 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.796515 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.799091 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.799797 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.799936 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.805024 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:55.805031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:55.805037 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:55.805085 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:55.809351 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.810467 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.810473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.810477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.810640 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.811118 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.811123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.811127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.811499 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:55.811668 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:55.812180 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 01:29:55.812433 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:29:55.815157 1996 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.160.61:37869: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4017e314eba74525b086cc1dcbd5e905"
member_type: NON_VOTER
last_known_addr {
host: "127.27.160.61"
port: 37869
}
attrs {
promote: true
}
}
dest_uuid: "5e268b00cbb64f1fa59b486526de6a63"
cas_config_opid_index: -1
I20260925 01:29:55.816267 1996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } }
I20260925 01:29:55.819321 2557 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4017e314eba74525b086cc1dcbd5e905 (127.27.160.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } }
I20260925 01:29:55.820396 2501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.60.keytab.
Entry for principal oryx/127.27.160.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.60.keytab.
I20260925 01:29:55.824611 2557 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.160.61:37869
I20260925 01:29:55.825151 2557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } }
I20260925 01:29:55.825407 2557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:55.825466 2459 master.cc:537] Master@127.27.160.61:37869 shutting down...
I20260925 01:29:55.825803 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.60:41287
--webserver_interface=127.27.160.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.60.keytab
--principal=oryx/127.27.160.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:33983,127.27.160.61:37869,127.27.160.60:41287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:29:55.831032 2501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Refusing update from remote peer 5e268b00cbb64f1fa59b486526de6a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 01:29:55.832751 2556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:29:55.836589 1983 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 01:29:55.839226 1983 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 -> Peer 4017e314eba74525b086cc1dcbd5e905 (127.27.160.61:37869): Couldn't send request to peer 4017e314eba74525b086cc1dcbd5e905. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 01:29:55.840454 2459 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:55.840916 2459 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:55.841153 2459 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: stopping tablet replica
I20260925 01:29:55.847728 2459 master.cc:552] Master@127.27.160.61:37869 shutdown complete.
I20260925 01:29:55.847831 2459 master_runner.cc:305] Clearing existing system tablet
I20260925 01:29:55.848191 2459 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:55.848297 2459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:55.850723 2459 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 01:29:55.852208 2562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:55.852562 2459 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:55.852890 2459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 01:29:55.852982 2459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
uuid: "4017e314eba74525b086cc1dcbd5e905"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "90808429901203228ac358f6cae3cd74"
server_key_iv: "f13ee50345052f62fca0d4fa98a547f0"
server_key_version: "encryptionkey@0"
I20260925 01:29:55.856446 2459 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:55.861295 2459 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 01:29:55.861478 2459 log.cc:1196] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 01:29:55.861871 2459 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Deleting consensus metadata
I20260925 01:29:55.862150 2459 master_runner.cc:315] Copying system tablet from 127.27.160.62:33983
I20260925 01:29:55.864517 2459 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: tablet copy: Beginning tablet copy session from remote peer at address 127.27.160.62:33983
I20260925 01:29:55.877141 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.865293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56898 (local address 127.27.160.62:33983)
0925 01:29:55.865507 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:55.865514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:55.866051 (+ 537us) server_negotiation.cc:407] Connection header received
0925 01:29:55.866270 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:55.866276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:55.866351 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:55.866455 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:55.867268 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.868012 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.869368 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.869783 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.869911 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.871663 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:55.871669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:55.871674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:55.871714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:55.874540 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.875153 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.875160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.875164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.875299 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:55.875650 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:55.875656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:55.875659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:55.875925 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:55.876082 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:55.876448 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 01:29:55.876681 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:29:55.877202 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:55.865293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:56898)
0925 01:29:55.865924 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:55.865943 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:55.866122 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:55.866484 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:55.866489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:55.866910 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:55.867117 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.867126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.868165 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.868170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.869232 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:55.869248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:55.869937 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:55.869943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:55.870146 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:55.870781 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:55.870798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:55.871462 (+ 664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:55.874672 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.874677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.874682 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.875032 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.875407 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:55.875412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:55.875416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:55.875546 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:55.876054 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:55.876058 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:55.876221 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 01:29:55.876360 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 01:29:55.876659 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":379,"threads_started":1}
I20260925 01:29:55.878314 2017 tablet_copy_service.cc:140] P 5e268b00cbb64f1fa59b486526de6a63: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4017e314eba74525b086cc1dcbd5e905 ({username='oryx', principal='oryx/127.27.160.61@KRBTEST.COM'} at 127.0.0.1:56898)
I20260925 01:29:55.878659 2017 tablet_copy_service.cc:161] P 5e268b00cbb64f1fa59b486526de6a63: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4017e314eba74525b086cc1dcbd5e905 at {username='oryx', principal='oryx/127.27.160.61@KRBTEST.COM'} at 127.0.0.1:56898: session id = 4017e314eba74525b086cc1dcbd5e905-00000000000000000000000000000000
I20260925 01:29:55.881892 2017 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 01:29:55.883725 2459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 01:29:55.888247 2459 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: tablet copy: Starting download of 0 data blocks...
I20260925 01:29:55.888561 2459 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: tablet copy: Starting download of 1 WAL segments...
I20260925 01:29:55.890988 2459 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 01:29:55.894414 2017 tablet_copy_service.cc:342] P 5e268b00cbb64f1fa59b486526de6a63: Request end of tablet copy session 4017e314eba74525b086cc1dcbd5e905-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.160.61@KRBTEST.COM'} at 127.0.0.1:56898
I20260925 01:29:55.894613 2017 tablet_copy_service.cc:433] P 5e268b00cbb64f1fa59b486526de6a63: ending tablet copy session 4017e314eba74525b086cc1dcbd5e905-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4017e314eba74525b086cc1dcbd5e905
I20260925 01:29:55.904302 2459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:55.908535 2569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:55.913522 2570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:55.918303 2572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:55.918573 2459 server_base.cc:1034] running on GCE node
I20260925 01:29:55.918879 2459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:55.919536 2459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:55.920742 2459 hybrid_clock.cc:648] HybridClock initialized: now 1790299795920687 us; error 38 us; skew 500 ppm
I20260925 01:29:55.920969 2459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:55 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299795, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:55.923058 2459 init.cc:377] Logged in from keytab as oryx/127.27.160.61@KRBTEST.COM (short username oryx)
I20260925 01:29:55.924332 2459 webserver.cc:460] Webserver started at http://127.27.160.61:42601/ using document root <none> and password file <none>
I20260925 01:29:55.924678 2459 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:55.924767 2459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:55.927160 2459 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 01:29:55.928911 2579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:55.929368 2459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 01:29:55.929519 2459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
uuid: "4017e314eba74525b086cc1dcbd5e905"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "90808429901203228ac358f6cae3cd74"
server_key_iv: "f13ee50345052f62fca0d4fa98a547f0"
server_key_version: "encryptionkey@0"
I20260925 01:29:55.929687 2459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:55.949251 2459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:55.952826 2459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:55.981837 2459 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.61:37869
I20260925 01:29:55.981860 2630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.61:37869 every 8 connection(s)
I20260925 01:29:55.982782 2459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
I20260925 01:29:55.990284 2631 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:29:55.993813 2631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Bootstrap starting.
I20260925 01:29:56.006705 2631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:29:56.007268 2631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: Bootstrap complete.
I20260925 01:29:56.008160 2631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } }
I20260925 01:29:56.008391 2631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4017e314eba74525b086cc1dcbd5e905, State: Initialized, Role: LEARNER
I20260925 01:29:56.008621 2631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } }
I20260925 01:29:56.009495 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } }
I20260925 01:29:56.009817 2631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:56.009963 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: This master's current role is: LEARNER
I20260925 01:29:56.018252 2645 catalog_manager.cc:1259] Loaded cluster ID: f4ff7f0d1de44a2d928d3ed9de76f417
I20260925 01:29:56.018388 2645 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: loading cluster ID for follower catalog manager: success
I20260925 01:29:56.020913 2645 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: acquiring CA information for follower catalog manager: success
I20260925 01:29:56.022933 2645 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:29:56.169562 2453 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
W20260925 01:29:56.300181 2559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:56.300628 2559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:56.300781 2559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:56.312513 2559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:29:56.312649 2559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:56.312731 2559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:56.312819 2559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:29:56.312880 2559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:29:56.328302 2559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:33983,127.27.160.61:37869,127.27.160.60:41287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.60.keytab
--principal=oryx/127.27.160.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.60:41287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.160.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:56.330083 2559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:56.331740 2559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:56.344753 2651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:56.344831 2653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:56.345377 2559 server_base.cc:1034] running on GCE node
W20260925 01:29:56.345954 2650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:56.347231 2559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:56.348713 2559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:56.349936 2559 hybrid_clock.cc:648] HybridClock initialized: now 1790299796349868 us; error 50 us; skew 500 ppm
I20260925 01:29:56.350333 2559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:29:56.353634 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.346175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.0.0.1:43472)
0925 01:29:56.346692 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.346719 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.346857 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.347274 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.347280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.347324 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:56.347552 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.347568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.349706 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.349710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.351590 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.351622 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.352771 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.352777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.352977 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.353034 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.353169 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.353310 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260925 01:29:56.353660 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.346344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.27.160.61:37869)
0925 01:29:56.346917 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.346925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.346951 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:29:56.347046 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.347050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.347118 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.347240 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:56.347747 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.349552 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.351750 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.352593 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.352753 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.352992 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.353193 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.353325 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":2144,"threads_started":1}
I20260925 01:29:56.355003 2606 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 01:29:56.355680 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e268b00cbb64f1fa59b486526de6a63. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } }
I20260925 01:29:56.355958 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: This master's current role is: LEARNER
Sep 25 01:29:56 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299796, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:56.357765 2559 init.cc:377] Logged in from keytab as oryx/127.27.160.60@KRBTEST.COM (short username oryx)
I20260925 01:29:56.359305 2559 webserver.cc:460] Webserver started at http://127.27.160.60:36081/ using document root <none> and password file <none>
I20260925 01:29:56.360037 2559 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:56.360181 2559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:56.360486 2559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:29:56.363617 2559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data/instance:
uuid: "c8dcccfd51814f85b39ff6635a86ff4f"
format_stamp: "Formatted at 2026-09-25 01:29:56 on dist-test-slave-h6hf"
server_key: "84b9a300af1c5f2304db33aa20fbba5b"
server_key_iv: "f9a1a2a29e1f0cacf3edd999ac9a0dbd"
server_key_version: "encryptionkey@0"
I20260925 01:29:56.364403 2559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal/instance:
uuid: "c8dcccfd51814f85b39ff6635a86ff4f"
format_stamp: "Formatted at 2026-09-25 01:29:56 on dist-test-slave-h6hf"
server_key: "84b9a300af1c5f2304db33aa20fbba5b"
server_key_iv: "f9a1a2a29e1f0cacf3edd999ac9a0dbd"
server_key_version: "encryptionkey@0"
I20260925 01:29:56.369915 2559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 01:29:56.373605 2663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:56.375150 2559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:29:56.375341 2559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
uuid: "c8dcccfd51814f85b39ff6635a86ff4f"
format_stamp: "Formatted at 2026-09-25 01:29:56 on dist-test-slave-h6hf"
server_key: "84b9a300af1c5f2304db33aa20fbba5b"
server_key_iv: "f9a1a2a29e1f0cacf3edd999ac9a0dbd"
server_key_version: "encryptionkey@0"
I20260925 01:29:56.375545 2559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:56.415269 2559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:56.419518 2559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:56.419822 2559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:56.439062 2559 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.60:41287
I20260925 01:29:56.439101 2714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.60:41287 every 8 connection(s)
I20260925 01:29:56.440452 2559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
I20260925 01:29:56.444108 2715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:56.446494 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 2559
I20260925 01:29:56.451645 2715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:29:56 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.60@KRBTEST.COM
Sep 25 01:29:56 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299796, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.60@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:29:56.471014 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.455234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.27.160.62:33983)
0925 01:29:56.456267 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.456273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.456740 (+ 467us) server_negotiation.cc:407] Connection header received
0925 01:29:56.456943 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.456947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.457011 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.457120 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.458257 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.459184 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.460501 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.461275 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.461437 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.464961 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.464970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.464977 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.465027 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.467941 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.468728 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.468739 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.468741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.468877 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.469355 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.469362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.469365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.469637 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.469839 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.470398 (+ 559us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.470613 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":705,"threads_started":1}
I20260925 01:29:56.471726 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.448761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.27.160.60:41287)
0925 01:29:56.449583 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.449595 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.449638 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:29:56.451239 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.451260 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.451672 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.451997 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.454376 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.456330 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.458097 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.458783 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.458908 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.464437 (+ 5529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.464446 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.464459 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.464488 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.467267 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.467959 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.467966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.467973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.468122 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.468542 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.468551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.468558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.469077 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.469436 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.469889 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.470336 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":389,"threads_started":1}
I20260925 01:29:56.471767 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.455238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:56912)
0925 01:29:56.456612 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.456653 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.456817 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.457203 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.457210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.457816 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:56.458101 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.458111 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.459327 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.459330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.460351 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.460362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.461492 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.461496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.461626 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.462276 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:56.462301 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:56.464750 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:56.468086 (+ 3336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.468101 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.468109 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.468591 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.468993 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.468999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.469003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.469229 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.469821 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:56.469837 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:56.470032 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.470255 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.470618 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"thread_start_us":974,"threads_started":1}
I20260925 01:29:56.474517 2715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:29:56 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299796, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.60@KRBTEST.COM for oryx/127.27.160.61@KRBTEST.COM
I20260925 01:29:56.488566 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.475372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.0.0.1:43474)
0925 01:29:56.475566 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.475585 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.475700 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.476007 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.476018 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.476470 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:56.476657 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.476668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.477671 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.477677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.479229 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.479244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.480052 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.480059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.480275 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.480878 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:56.480904 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:56.483246 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:56.486225 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.486231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.486235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.486637 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.486998 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.487003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.487006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.487147 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.487644 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:56.487649 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:56.487790 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.487924 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.488228 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 01:29:56.488596 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.475394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.27.160.61:37869)
0925 01:29:56.475566 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.475570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.475651 (+ 81us) server_negotiation.cc:407] Connection header received
0925 01:29:56.475816 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.475820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.475882 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.475963 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.476800 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.477546 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.479373 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.479908 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.480000 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.483437 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.483442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.483445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.483475 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.486093 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.486751 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.486757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.486760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.486888 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.487250 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.487257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.487262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.487520 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.487679 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.488025 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.488243 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:29:56.489689 2715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:29:56 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299796, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.60@KRBTEST.COM for oryx/127.27.160.60@KRBTEST.COM
I20260925 01:29:56.501462 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.490467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.27.160.60:41287)
0925 01:29:56.490607 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.490613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.490628 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:29:56.490696 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.490700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.490785 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.490865 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.491647 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.492382 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.493528 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.493921 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.494046 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.496732 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.496736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.496739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.496765 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.499031 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.499610 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.499618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.499621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.499723 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.500078 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.500084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.500087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.500342 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.500485 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.500799 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.501151 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":43}
I20260925 01:29:56.501471 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.490314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:41287 (local address 127.0.0.1:50732)
0925 01:29:56.490481 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.490512 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.490607 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.490881 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.490885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.491306 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:56.491506 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.491516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.492501 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.492508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.493405 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.493417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.494050 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.494064 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.494271 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.494726 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:56.494741 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:56.496564 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:56.499161 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.499168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.499172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.499499 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.499822 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.499827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.499831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.499971 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.500450 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:56.500454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:56.500601 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.500730 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.500992 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260925 01:29:56.506283 2715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Bootstrap starting.
I20260925 01:29:56.509369 2715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:56.510444 2715 log.cc:824] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:56.513247 2715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: No bootstrap required, opened a new log
I20260925 01:29:56.520740 2715 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } }
I20260925 01:29:56.521268 2715 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:56.521379 2715 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8dcccfd51814f85b39ff6635a86ff4f, State: Initialized, Role: FOLLOWER
I20260925 01:29:56.522015 2715 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } }
I20260925 01:29:56.522936 2722 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } } }
I20260925 01:29:56.523226 2722 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:29:56.524410 2715 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:29:56.534315 2733 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:29:56.534494 2733 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:29:56.538610 2559 master_runner.cc:428] Detected that this master c8dcccfd51814f85b39ff6635a86ff4f is joining an existing cluster
I20260925 01:29:56.538792 2559 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.160.60:41287
I20260925 01:29:56.548785 2722 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:56.549166 2722 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } }
I20260925 01:29:56.550890 2722 leader_election.cc:288] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e268b00cbb64f1fa59b486526de6a63 (127.27.160.62:33983), 4017e314eba74525b086cc1dcbd5e905 (127.27.160.61:37869)
I20260925 01:29:56.552001 2606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4017e314eba74525b086cc1dcbd5e905" is_pre_election: true
I20260925 01:29:56.553452 2007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e268b00cbb64f1fa59b486526de6a63" is_pre_election: true
I20260925 01:29:56.555085 2666 leader_election.cc:302] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8dcccfd51814f85b39ff6635a86ff4f; no voters: 4017e314eba74525b086cc1dcbd5e905, 5e268b00cbb64f1fa59b486526de6a63
I20260925 01:29:56.555976 2722 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 01:29:56.560775 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.542832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.27.160.62:33983)
0925 01:29:56.543298 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.543305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.543973 (+ 668us) server_negotiation.cc:407] Connection header received
0925 01:29:56.544325 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.544335 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.544446 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.544555 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.546581 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.548133 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.550263 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.550893 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.551032 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.553789 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.553799 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.553806 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.553848 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.557383 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.558199 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.558205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.558210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.558386 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.558847 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.558852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.558855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.559243 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.559432 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.560176 (+ 744us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.560415 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
I20260925 01:29:56.561039 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.542828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:56914)
0925 01:29:56.543835 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.543861 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.544133 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.544885 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.544893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.545850 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:56.546344 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.546362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.548427 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.548436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.550091 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.550104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.551349 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.551355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.551542 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.552289 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:56.552311 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:56.553544 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:56.557561 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.557566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.557573 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.558043 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.558523 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.558529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.558532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.558715 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.559498 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:56.559504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:56.559760 (+ 256us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.560054 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.560477 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":658,"threads_started":1}
I20260925 01:29:56.572127 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.544304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.27.160.60:41287)
0925 01:29:56.544519 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.544525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.554957 (+ 10432us) server_negotiation.cc:407] Connection header received
0925 01:29:56.555229 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.555240 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.555311 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.555390 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.556914 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.558040 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.559871 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.560633 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.560725 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.563550 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.563557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.563563 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.563600 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.568028 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.568936 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.568942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.568946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.569159 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.569608 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.569613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.569617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.570037 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.570190 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.570772 (+ 582us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.571541 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:29:56.574852 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.542823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.27.160.61:37869)
0925 01:29:56.543046 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.543056 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.554087 (+ 11031us) server_negotiation.cc:407] Connection header received
0925 01:29:56.554273 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.554280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.554367 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.554480 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.556136 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.557252 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.559020 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.559709 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.559851 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.565207 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.565214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.565219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.565264 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.570536 (+ 5272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.571659 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.571667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.571674 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.571875 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.572394 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.572399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.572418 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.572841 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.573036 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.573979 (+ 943us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.574355 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:29:56.576680 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.543990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:41287 (local address 127.0.0.1:50746)
0925 01:29:56.554856 (+ 10866us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.554878 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.555072 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.555584 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.555591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.556406 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:56.556751 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.556763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.558237 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.558240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.559732 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.559747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.561051 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.561056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.561271 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.561949 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:56.562012 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:56.563335 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:56.568187 (+ 4852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.568193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.568197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.568790 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.569312 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.569317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.569321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.569485 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.570275 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:56.570280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:56.570464 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.571000 (+ 536us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.573549 (+ 2549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10705,"mutex_wait_us":404,"thread_start_us":10261,"threads_started":1}
I20260925 01:29:56.576715 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.543477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.0.0.1:43480)
0925 01:29:56.553977 (+ 10500us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.554000 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.554149 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.554707 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.554713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.555491 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:56.555915 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.555935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.557415 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.557418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.558839 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.558867 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.562445 (+ 3578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.562456 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.562664 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.563330 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:56.563357 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:56.564933 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:56.570748 (+ 5815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.570756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.570762 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.571475 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.572055 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.572063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.572069 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.572269 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.573168 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:56.573176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:56.573641 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.573811 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.574399 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10340,"mutex_wait_us":2,"thread_start_us":12764,"threads_started":1}
I20260925 01:29:56.578800 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 2058
Sep 25 01:29:56 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299793, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.62@KRBTEST.COM for oryx/127.27.160.60@KRBTEST.COM
I20260925 01:29:56.591269 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.566944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:41287 (local address 127.0.0.1:50752)
0925 01:29:56.567266 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.567291 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.567427 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.572958 (+ 5531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.572965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.573565 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:56.573806 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.573822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.575674 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.575680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.577217 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.577234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.579023 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.579031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.579229 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.581280 (+ 2051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:56.581310 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:56.584253 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:56.588228 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.588233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.588237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.588805 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.589265 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.589270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.589276 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.589486 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.590187 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:56.590195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:56.590393 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.590562 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.590865 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
Sep 25 01:29:56 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.1@KRBTEST.COM
I20260925 01:29:56.591756 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.567027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.27.160.60:41287)
0925 01:29:56.572453 (+ 5426us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.572459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.572519 (+ 60us) server_negotiation.cc:407] Connection header received
0925 01:29:56.572629 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.572636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.572732 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.574022 (+ 1290us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.574463 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.575521 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.577967 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.578784 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.579616 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.584465 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.584472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.584476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.584512 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.588098 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.588955 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.588961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.588965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.589145 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.589638 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.589646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.589651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.590039 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.590187 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.590705 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.590948 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5304,"thread_start_us":9021,"threads_started":1}
I20260925 01:29:56.593230 1996 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.160.60:41287: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f"
member_type: NON_VOTER
last_known_addr {
host: "127.27.160.60"
port: 41287
}
attrs {
promote: true
}
}
dest_uuid: "5e268b00cbb64f1fa59b486526de6a63"
cas_config_opid_index: 5
I20260925 01:29:56.594375 1996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } }
I20260925 01:29:56.595986 2655 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c8dcccfd51814f85b39ff6635a86ff4f (127.27.160.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.596980 2690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:56.601876 2690 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Refusing update from remote peer 5e268b00cbb64f1fa59b486526de6a63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 01:29:56.612680 2606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 LEARNER]: Refusing update from remote peer 5e268b00cbb64f1fa59b486526de6a63: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 01:29:56.613068 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.580110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.27.160.1:40581)
0925 01:29:56.580699 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.580709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.580739 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:29:56.580811 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.580819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.581137 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.583078 (+ 1941us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.583316 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.584496 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.586443 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.587441 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.587591 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.592928 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.592946 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.592968 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.593029 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.603791 (+ 10762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.606779 (+ 2988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.606790 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.606794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.606983 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.607968 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.607976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.607981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.608500 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.610753 (+ 2253us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.610934 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.611233 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":3308,"threads_started":1}
I20260925 01:29:56.615140 2181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:56.618271 2655 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.160.60:41287
I20260925 01:29:56.618387 2745 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:56.619084 2559 master.cc:537] Master@127.27.160.60:41287 shutting down...
I20260925 01:29:56.618964 2655 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 01:29:56.619943 2746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.622844 2746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:56.632195 2606 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c8dcccfd51814f85b39ff6635a86ff4f (127.27.160.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
W20260925 01:29:56.634305 1982 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 -> Peer c8dcccfd51814f85b39ff6635a86ff4f (127.27.160.60:41287): Couldn't send request to peer c8dcccfd51814f85b39ff6635a86ff4f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 01:29:56.636759 2746 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: attempting to promote NON_VOTER 4017e314eba74525b086cc1dcbd5e905 to VOTER
I20260925 01:29:56.637331 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: NON_VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: true } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.637617 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: This master's current role is: LEARNER
I20260925 01:29:56.637728 2746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } }
I20260925 01:29:56.639875 2606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 LEARNER]: Refusing update from remote peer 5e268b00cbb64f1fa59b486526de6a63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 01:29:56.640491 2747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260925 01:29:56.640954 1982 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 -> Peer c8dcccfd51814f85b39ff6635a86ff4f (127.27.160.60:41287): Couldn't send request to peer c8dcccfd51814f85b39ff6635a86ff4f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 01:29:56.642949 2655 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4017e314eba74525b086cc1dcbd5e905 (127.27.160.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.643751 2606 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4017e314eba74525b086cc1dcbd5e905 (127.27.160.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.647629 2746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.647926 2746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:56.647658 2745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.648316 2745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:56.648893 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.649164 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:29:56.658365 2559 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:56.658840 2559 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:56.659066 2559 tablet_replica.cc:330] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: stopping tablet replica
I20260925 01:29:56.666584 2559 master.cc:552] Master@127.27.160.60:41287 shutdown complete.
I20260925 01:29:56.666697 2559 master_runner.cc:305] Clearing existing system tablet
I20260925 01:29:56.667114 2559 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:56.667223 2559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:56.670503 2559 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 01:29:56.672335 2753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:56.672791 2559 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:56.673412 2559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 01:29:56.673600 2559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
uuid: "c8dcccfd51814f85b39ff6635a86ff4f"
format_stamp: "Formatted at 2026-09-25 01:29:56 on dist-test-slave-h6hf"
server_key: "84b9a300af1c5f2304db33aa20fbba5b"
server_key_iv: "f9a1a2a29e1f0cacf3edd999ac9a0dbd"
server_key_version: "encryptionkey@0"
I20260925 01:29:56.679214 2559 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:56.687127 2559 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 01:29:56.687440 2559 log.cc:1196] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 01:29:56.688066 2559 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Deleting consensus metadata
I20260925 01:29:56.688416 2559 master_runner.cc:315] Copying system tablet from 127.27.160.62:33983
I20260925 01:29:56.691210 2559 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: tablet copy: Beginning tablet copy session from remote peer at address 127.27.160.62:33983
I20260925 01:29:56.717497 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.692216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.27.160.62:33983)
0925 01:29:56.692476 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:56.692483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:56.701845 (+ 9362us) server_negotiation.cc:407] Connection header received
0925 01:29:56.702059 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:56.702065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:56.702155 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:56.702271 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:56.703468 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.704651 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.706764 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.707388 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.707537 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.710403 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:56.710409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:56.710415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:56.710478 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:56.714105 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.715105 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.715111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.715114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.715285 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:56.715848 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:56.715853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:56.715857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:56.716200 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:56.716343 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:56.716874 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 01:29:56.717099 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260925 01:29:56.718089 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:56.692117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:56920)
0925 01:29:56.701659 (+ 9542us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:56.701687 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:56.701890 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:56.702329 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:56.702339 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:56.702944 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:56.703257 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.703271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.704826 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.704833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.706582 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:56.706600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:56.707571 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:56.707580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:56.707795 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:56.708858 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:56.708884 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:56.710202 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:56.714290 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.714298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.714303 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.714883 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.715498 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:56.715504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:56.715509 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:56.715695 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:56.716329 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:56.716334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:56.716562 (+ 228us) client_negotiation.cc:769] Sending connection context
0925 01:29:56.716720 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 01:29:56.717081 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9354,"thread_start_us":9202,"threads_started":1}
I20260925 01:29:56.718792 2017 tablet_copy_service.cc:140] P 5e268b00cbb64f1fa59b486526de6a63: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c8dcccfd51814f85b39ff6635a86ff4f ({username='oryx', principal='oryx/127.27.160.60@KRBTEST.COM'} at 127.0.0.1:56920)
I20260925 01:29:56.718967 2017 tablet_copy_service.cc:161] P 5e268b00cbb64f1fa59b486526de6a63: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c8dcccfd51814f85b39ff6635a86ff4f at {username='oryx', principal='oryx/127.27.160.60@KRBTEST.COM'} at 127.0.0.1:56920: session id = c8dcccfd51814f85b39ff6635a86ff4f-00000000000000000000000000000000
I20260925 01:29:56.722647 2017 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 01:29:56.724669 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 01:29:56.731458 2559 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: tablet copy: Starting download of 0 data blocks...
I20260925 01:29:56.731781 2559 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: tablet copy: Starting download of 1 WAL segments...
I20260925 01:29:56.734316 2559 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 01:29:56.738723 2017 tablet_copy_service.cc:342] P 5e268b00cbb64f1fa59b486526de6a63: Request end of tablet copy session c8dcccfd51814f85b39ff6635a86ff4f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.160.60@KRBTEST.COM'} at 127.0.0.1:56920
I20260925 01:29:56.738864 2017 tablet_copy_service.cc:433] P 5e268b00cbb64f1fa59b486526de6a63: ending tablet copy session c8dcccfd51814f85b39ff6635a86ff4f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c8dcccfd51814f85b39ff6635a86ff4f
I20260925 01:29:56.748868 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 2058
I20260925 01:29:56.752472 2559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:56.755360 2760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:56.762722 2761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:56.769837 2763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:56.771193 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:40581
--local_ip_for_outbound_sockets=127.27.160.1
--tserver_master_addrs=127.27.160.62:33983
--webserver_port=33737
--webserver_interface=127.27.160.1
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab
--principal=oryx/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869,127.27.160.60:41287 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:29:56.771541 2559 server_base.cc:1034] running on GCE node
I20260925 01:29:56.771865 2559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:56.772580 2559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:56.773772 2559 hybrid_clock.cc:648] HybridClock initialized: now 1790299796773726 us; error 39 us; skew 500 ppm
I20260925 01:29:56.774019 2559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:56 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299796, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:56.775858 2559 init.cc:377] Logged in from keytab as oryx/127.27.160.60@KRBTEST.COM (short username oryx)
I20260925 01:29:56.778928 2559 webserver.cc:460] Webserver started at http://127.27.160.60:37765/ using document root <none> and password file <none>
I20260925 01:29:56.779325 2559 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:56.779436 2559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:56.782326 2559 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 01:29:56.784111 2771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:56.784593 2559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 01:29:56.784735 2559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
uuid: "c8dcccfd51814f85b39ff6635a86ff4f"
format_stamp: "Formatted at 2026-09-25 01:29:56 on dist-test-slave-h6hf"
server_key: "84b9a300af1c5f2304db33aa20fbba5b"
server_key_iv: "f9a1a2a29e1f0cacf3edd999ac9a0dbd"
server_key_version: "encryptionkey@0"
I20260925 01:29:56.784933 2559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:56.794076 2559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:56.798262 2559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:56.843712 2559 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.60:41287
I20260925 01:29:56.843732 2822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.60:41287 every 8 connection(s)
I20260925 01:29:56.845037 2559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
I20260925 01:29:56.856796 2823 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:29:56.862663 2823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Bootstrap starting.
I20260925 01:29:56.877435 2823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:29:56.877969 2823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Bootstrap complete.
I20260925 01:29:56.878955 2823 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } }
I20260925 01:29:56.879215 2823 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c8dcccfd51814f85b39ff6635a86ff4f, State: Initialized, Role: LEARNER
I20260925 01:29:56.879492 2823 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } }
I20260925 01:29:56.880425 2826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:56.880792 2823 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:29:56.880923 2826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: This master's current role is: LEARNER
I20260925 01:29:56.896278 2837 catalog_manager.cc:1259] Loaded cluster ID: f4ff7f0d1de44a2d928d3ed9de76f417
I20260925 01:29:56.896416 2837 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: loading cluster ID for follower catalog manager: success
I20260925 01:29:56.899060 2837 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: acquiring CA information for follower catalog manager: success
I20260925 01:29:56.900904 2837 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:29:57.124378 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.101517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:41287 (local address 127.0.0.1:50764)
0925 01:29:57.117158 (+ 15641us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:57.117187 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:57.117374 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:57.117837 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:57.117842 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:57.117870 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:57.118124 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.118136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.120205 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.120210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:57.122145 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.122160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.123803 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.123808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:57.123935 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.123960 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:29:57.124036 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 01:29:57.124137 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15475,"thread_start_us":15953,"threads_started":1}
I20260925 01:29:57.124632 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.101781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.27.160.60:41287)
0925 01:29:57.117330 (+ 15549us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.117340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.117371 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:29:57.117543 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.117550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.117639 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.117783 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:57.118310 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.120020 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.122297 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.123640 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.123789 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.124064 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:57.124122 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 01:29:57.124263 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15376,"thread_start_us":15587,"threads_started":1}
I20260925 01:29:57.125488 2798 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 01:29:57.126140 2826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e268b00cbb64f1fa59b486526de6a63. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:57.126434 2826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: This master's current role is: LEARNER
W20260925 01:29:57.200809 2769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:57.201116 2769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:57.201303 2769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:57.208519 2769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:57.208662 2769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:57.208825 2769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:29:57.219060 2769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab
--principal=oryx/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:40581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=33737
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869,127.27.160.60:41287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:57.220443 2769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:57.221961 2769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:57.235327 2844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:57.235391 2845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:57.235261 2847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:57.236061 2769 server_base.cc:1034] running on GCE node
I20260925 01:29:57.236611 2769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:57.237553 2769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:57.238749 2769 hybrid_clock.cc:648] HybridClock initialized: now 1790299797238695 us; error 41 us; skew 500 ppm
I20260925 01:29:57.239023 2769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:57 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299797, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:57.242012 2769 init.cc:377] Logged in from keytab as oryx/127.27.160.1@KRBTEST.COM (short username oryx)
I20260925 01:29:57.243324 2769 webserver.cc:460] Webserver started at http://127.27.160.1:33737/ using document root <none> and password file <none>
I20260925 01:29:57.243957 2769 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:57.244076 2769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:57.249547 2769 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:29:57.252111 2854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:57.253465 2769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:29:57.253608 2769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "764191dcd7ec42758b4181b222acd471"
format_stamp: "Formatted at 2026-09-25 01:29:54 on dist-test-slave-h6hf"
server_key: "b4f3a762307699fa861af098681ebf71"
server_key_iv: "b460a89fe7a24cadd6537f9c74746743"
server_key_version: "encryptionkey@0"
I20260925 01:29:57.254524 2769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:57.272224 2769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:57.276289 2769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:57.276530 2769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:57.277544 2769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:57.278913 2769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:57.278996 2769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:57.279088 2769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:57.279141 2769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:57.312937 2769 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:40581
I20260925 01:29:57.313035 2966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:40581 every 8 connection(s)
I20260925 01:29:57.314594 2769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:29:57.320616 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 2769
I20260925 01:29:57.321048 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 2192
Sep 25 01:29:57 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299797, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.61@KRBTEST.COM
Sep 25 01:29:57 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299797, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.60@KRBTEST.COM
I20260925 01:29:57.346613 2839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.317586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:51197 (local address 127.27.160.60:41287)
0925 01:29:57.317787 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.317793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.319404 (+ 1611us) server_negotiation.cc:407] Connection header received
0925 01:29:57.327977 (+ 8573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.327986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.328108 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.328196 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:57.330397 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.331842 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.333102 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.334041 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.334194 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.339336 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:57.339346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:57.339353 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:57.339408 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:57.343198 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:57.344047 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:57.344053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:57.344056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:57.344193 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:57.344626 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:57.344631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:57.344635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:57.344985 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:57.345107 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:57.346023 (+ 916us) server_negotiation.cc:299] Negotiation successful
0925 01:29:57.346254 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:29:57.346642 2971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.317632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:54853 (local address 127.27.160.61:37869)
0925 01:29:57.318108 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.318117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.320779 (+ 2662us) server_negotiation.cc:407] Connection header received
0925 01:29:57.322696 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.322705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.322826 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.322939 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:57.325503 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.326918 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.331162 (+ 4244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.332000 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.332127 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.338507 (+ 6380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:57.338515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:57.338521 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:57.338579 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:57.342350 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:57.343406 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:57.343413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:57.343417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:57.343588 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:57.344225 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:57.344230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:57.344233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:57.344648 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:57.344780 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:57.346005 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0925 01:29:57.346238 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":969,"threads_started":1}
I20260925 01:29:57.349176 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.318300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.27.160.1:54853)
0925 01:29:57.319219 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:57.319285 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:57.322254 (+ 2969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:57.323432 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:57.323446 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:57.324329 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:57.325270 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.325300 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.327159 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.327167 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:57.330968 (+ 3801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.330997 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.332509 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.332517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:57.332905 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.334022 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:57.334065 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:57.338123 (+ 4058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:57.342548 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:57.342560 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:57.342581 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:57.343274 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:57.343735 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:57.343740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:57.343743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:57.344100 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:57.344876 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:57.344890 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:57.345588 (+ 698us) client_negotiation.cc:769] Sending connection context
0925 01:29:57.347378 (+ 1790us) client_negotiation.cc:240] Negotiation successful
0925 01:29:57.347701 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"spinlock_wait_cycles":29184,"thread_start_us":358,"threads_started":1}
I20260925 01:29:57.349205 2973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.318300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:41287 (local address 127.27.160.1:51197)
0925 01:29:57.319219 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:57.319285 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:57.327658 (+ 8373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:57.328486 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:57.328493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:57.329944 (+ 1451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:57.330239 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.330250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.331979 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.331984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:57.332956 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.332967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.334497 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.334507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:57.334810 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.335506 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:57.335542 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:57.339039 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:57.343365 (+ 4326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:57.343372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:57.343377 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:57.343905 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:57.344339 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:57.344343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:57.344347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:57.344487 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:57.345212 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:57.345216 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:57.345543 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 01:29:57.346832 (+ 1289us) client_negotiation.cc:240] Negotiation successful
0925 01:29:57.347483 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":48,"spinlock_wait_cycles":50048,"thread_start_us":433,"threads_started":1}
Sep 25 01:29:57 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299797, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:29:57.351253 2967 heartbeater.cc:347] Connected to a master server at 127.27.160.60:41287
I20260925 01:29:57.351565 2967 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:57.352250 2967 heartbeater.cc:511] Master 127.27.160.60:41287 requested a full tablet report, sending...
I20260925 01:29:57.351289 2969 heartbeater.cc:347] Connected to a master server at 127.27.160.61:37869
I20260925 01:29:57.353082 2969 heartbeater.cc:464] Registering TS with master...
Sep 25 01:29:57 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.2@KRBTEST.COM
I20260925 01:29:57.353529 2969 heartbeater.cc:511] Master 127.27.160.61:37869 requested a full tablet report, sending...
I20260925 01:29:57.354799 2788 ts_manager.cc:194] Registered new tserver with Master: 764191dcd7ec42758b4181b222acd471 (127.27.160.1:40581)
I20260925 01:29:57.355249 2596 ts_manager.cc:194] Registered new tserver with Master: 764191dcd7ec42758b4181b222acd471 (127.27.160.1:40581)
I20260925 01:29:57.359644 2974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.319780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.1:35553)
0925 01:29:57.329865 (+ 10085us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:57.329888 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:57.330031 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:57.342157 (+ 12126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:57.342164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:57.342757 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:57.343144 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.343156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.344681 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.344689 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:57.346231 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.346244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.347258 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.347264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:57.347437 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.348073 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:57.348105 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:57.351001 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:57.357091 (+ 6090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:57.357096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:57.357100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:57.357637 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:57.358025 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:57.358034 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:57.358038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:57.358226 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:57.358733 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:57.358740 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:57.358898 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 01:29:57.359003 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:29:57.359307 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9921,"thread_start_us":9734,"threads_started":1}
I20260925 01:29:57.359751 2975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.320178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:35553 (local address 127.27.160.62:33983)
0925 01:29:57.341774 (+ 21596us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.341782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.341803 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:29:57.341864 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.341871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.341948 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.343298 (+ 1350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:57.343461 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.344528 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.346414 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.347034 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.349303 (+ 2269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.353328 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:57.353338 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:57.353343 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:57.353386 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:57.356955 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:57.357761 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:57.357764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:57.357767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:57.357883 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:57.358332 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:57.358336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:57.358337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:57.358620 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:57.358762 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:57.359218 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 01:29:57.359415 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21453,"thread_start_us":19609,"threads_started":1}
I20260925 01:29:57.360610 2968 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:29:57.360756 2968 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:57.361233 2968 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
I20260925 01:29:57.361342 2977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.322197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.27.160.2:36355)
0925 01:29:57.333790 (+ 11593us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.333798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.340640 (+ 6842us) server_negotiation.cc:407] Connection header received
0925 01:29:57.340746 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.340758 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.341112 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.341330 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:57.343493 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.345084 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.347129 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.347938 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.348076 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.354438 (+ 6362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:57.354453 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:57.354468 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:57.354524 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:57.357972 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:57.358851 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:57.358858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:57.358860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:57.358982 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:57.359405 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:57.359409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:57.359413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:57.359820 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:57.360095 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:57.360668 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 01:29:57.360924 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11404,"thread_start_us":11428,"threads_started":1}
I20260925 01:29:57.362313 1996 ts_manager.cc:194] Re-registered known tserver with Master: 764191dcd7ec42758b4181b222acd471 (127.27.160.1:40581)
I20260925 01:29:57.362633 2315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:57.363480 1996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:35553
I20260925 01:29:57.448506 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 2192
I20260925 01:29:57.465411 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:36355
--local_ip_for_outbound_sockets=127.27.160.2
--tserver_master_addrs=127.27.160.62:33983
--webserver_port=43655
--webserver_interface=127.27.160.2
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab
--principal=oryx/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869,127.27.160.60:41287 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:29:57.490900 2747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: NON_VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: true } } }
I20260925 01:29:57.491261 2747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:57.502087 2747 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: attempting to promote NON_VOTER c8dcccfd51814f85b39ff6635a86ff4f to VOTER
I20260925 01:29:57.502908 2747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } }
I20260925 01:29:57.504866 2798 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 LEARNER]: Refusing update from remote peer 5e268b00cbb64f1fa59b486526de6a63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 01:29:57.505277 2606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Refusing update from remote peer 5e268b00cbb64f1fa59b486526de6a63: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 01:29:57.505501 2980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 01:29:57.505879 2747 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 01:29:57.508951 2606 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8dcccfd51814f85b39ff6635a86ff4f (127.27.160.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:29:57.508273 2747 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8dcccfd51814f85b39ff6635a86ff4f (127.27.160.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:29:57.520045 2985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:29:57.520412 2981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:29:57.520840 2981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:57.520880 2985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:29:57.521255 2981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:29:57.521548 2981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:57.524938 2981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:29:57.525287 2981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:29:57.525637 2798 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c8dcccfd51814f85b39ff6635a86ff4f (127.27.160.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:29:57.528816 2826 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e268b00cbb64f1fa59b486526de6a63" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:29:57.529136 2826 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: This master's current role is: FOLLOWER
W20260925 01:29:57.821563 2979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:57.821863 2979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:57.821969 2979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:57.829396 2979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:57.829523 2979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:57.829640 2979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:29:57.839767 2979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab
--principal=oryx/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:36355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=43655
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869,127.27.160.60:41287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:57.841554 2979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:57.843273 2979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:57.858616 2990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:57.858593 2991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:57.858592 2993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:57.859640 2979 server_base.cc:1034] running on GCE node
I20260925 01:29:57.860392 2979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:57.861446 2979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:57.862727 2979 hybrid_clock.cc:648] HybridClock initialized: now 1790299797862669 us; error 48 us; skew 500 ppm
I20260925 01:29:57.863102 2979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:57 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299797, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:57.866569 2979 init.cc:377] Logged in from keytab as oryx/127.27.160.2@KRBTEST.COM (short username oryx)
I20260925 01:29:57.868111 2979 webserver.cc:460] Webserver started at http://127.27.160.2:43655/ using document root <none> and password file <none>
I20260925 01:29:57.868876 2979 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:57.869041 2979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:57.874992 2979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:29:57.877789 3000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:57.879241 2979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:29:57.879423 2979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "22c8f4062454451b8404bb1cf3f389fc"
format_stamp: "Formatted at 2026-09-25 01:29:54 on dist-test-slave-h6hf"
server_key: "5843ae7e2a684a41220c4fcf75c1e3ae"
server_key_iv: "f0b83e84371f8c692cc726e90cb410c6"
server_key_version: "encryptionkey@0"
I20260925 01:29:57.880420 2979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:57.912853 2979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:57.917624 2979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:57.917909 2979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:57.918889 2979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:57.920252 2979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:57.920373 2979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:57.920511 2979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:57.920595 2979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:57.952637 2979 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:36355
I20260925 01:29:57.952694 3112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:36355 every 8 connection(s)
I20260925 01:29:57.954201 2979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:29:57.955291 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 2979
I20260925 01:29:57.955663 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 2325
Sep 25 01:29:57 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.3@KRBTEST.COM
Sep 25 01:29:58 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299797, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.60@KRBTEST.COM
Sep 25 01:29:58 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299797, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.61@KRBTEST.COM
I20260925 01:29:58.006213 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.956946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.27.160.3:33863)
0925 01:29:57.979256 (+ 22310us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.979270 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.980195 (+ 925us) server_negotiation.cc:407] Connection header received
0925 01:29:57.980537 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.980550 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.980982 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.981219 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:57.984494 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.986796 (+ 2302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.989403 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.990665 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.990803 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:57.998346 (+ 7543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:57.998363 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:57.998379 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:57.998426 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.002415 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.003359 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.003370 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.003376 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.003570 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.004061 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.004066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.004069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.004485 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.004786 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.005442 (+ 656us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.005753 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22126,"thread_start_us":21982,"threads_started":1}
Sep 25 01:29:58 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299797, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:29:58.008522 2448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:58.015225 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.973067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:54043 (local address 127.27.160.61:37869)
0925 01:29:57.976665 (+ 3598us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.976674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.987291 (+ 10617us) server_negotiation.cc:407] Connection header received
0925 01:29:57.994530 (+ 7239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.994540 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.994640 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.994735 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:57.998294 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.999428 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.001189 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.001677 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.001795 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.007400 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.007406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.007411 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.007466 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.010880 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.011686 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.011692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.011696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.011822 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.012244 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.012249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.012252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.012679 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.012870 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.014419 (+ 1549us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.014704 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3423,"thread_start_us":3273,"threads_started":1}
I20260925 01:29:58.017773 3119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.972542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.27.160.2:54043)
0925 01:29:57.980615 (+ 8073us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:57.980635 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:57.994050 (+ 13415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:57.995193 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:57.995205 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:57.996338 (+ 1133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:57.997458 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.997483 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.999558 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.999565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.001024 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.001037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.001925 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.001934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.002274 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.003247 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:58.003285 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:58.007162 (+ 3877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:58.011014 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.011025 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.011043 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.011574 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.011932 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.011936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.011940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.012128 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.012970 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:58.012979 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:58.013743 (+ 764us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.015533 (+ 1790us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.016129 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7918,"mutex_wait_us":110,"spinlock_wait_cycles":15744,"thread_start_us":5047,"threads_started":1}
I20260925 01:29:58.017827 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.957257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:35189 (local address 127.27.160.60:41287)
0925 01:29:57.972833 (+ 15576us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.972841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.979445 (+ 6604us) server_negotiation.cc:407] Connection header received
0925 01:29:57.994401 (+ 14956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.994407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.994501 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.994612 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:57.997647 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.998594 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.001550 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.002231 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.002357 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.008151 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.008157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.008161 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.008205 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.012449 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.015027 (+ 2578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.015033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.015036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.015227 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.015824 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.015830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.015833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.016251 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.016391 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.016926 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.017444 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15418,"thread_start_us":15577,"threads_started":1}
I20260925 01:29:58.018869 3121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.964028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:41287 (local address 127.27.160.2:35189)
0925 01:29:57.978178 (+ 14150us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:57.978251 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:57.994050 (+ 15799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:57.995433 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:57.995437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:57.996337 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:57.997454 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.997476 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:57.998813 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:57.998818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.001391 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.001405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.002585 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.002592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.002773 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.003397 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:58.003423 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:58.007941 (+ 4518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:58.012719 (+ 4778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.012727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.012733 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.014759 (+ 2026us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.015392 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.015396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.015399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.015691 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.016452 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:58.016458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:58.016633 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.018331 (+ 1698us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.018590 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13716,"mutex_wait_us":99,"thread_start_us":15461,"threads_started":1}
I20260925 01:29:58.019927 3115 heartbeater.cc:347] Connected to a master server at 127.27.160.61:37869
I20260925 01:29:58.020205 3113 heartbeater.cc:347] Connected to a master server at 127.27.160.60:41287
I20260925 01:29:58.020298 3115 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:58.020358 3113 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:58.021610 3113 heartbeater.cc:511] Master 127.27.160.60:41287 requested a full tablet report, sending...
I20260925 01:29:58.023172 3124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.980257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:38793 (local address 127.27.160.62:33983)
0925 01:29:57.991887 (+ 11630us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:57.991894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:57.992625 (+ 731us) server_negotiation.cc:407] Connection header received
0925 01:29:57.994653 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:57.994661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:57.994761 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:57.994868 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:57.998537 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.000836 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.004105 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.004763 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.004915 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.015011 (+ 10096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.015042 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.015048 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.015091 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.019921 (+ 4830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.021062 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.021069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.021072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.021288 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.021690 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.021697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.021702 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.022048 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.022200 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.022595 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.022810 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11482,"thread_start_us":11636,"threads_started":1}
I20260925 01:29:58.023200 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:57.980144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.2:38793)
0925 01:29:57.992418 (+ 12274us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:57.992439 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:57.994463 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:57.995196 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:57.995208 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:57.998015 (+ 2807us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:57.998348 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:57.998364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.001017 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.001022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.003925 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.003941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.004957 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.004964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.005215 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.005815 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:58.005840 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:58.014803 (+ 8963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:58.020122 (+ 5319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.020126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.020130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.020883 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.021394 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.021400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.021404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.021566 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.022161 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:58.022166 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:58.022306 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.022474 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.022783 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12147,"thread_start_us":11994,"threads_started":1}
I20260925 01:29:58.023514 2788 ts_manager.cc:194] Registered new tserver with Master: 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2:36355)
I20260925 01:29:58.024086 3115 heartbeater.cc:511] Master 127.27.160.61:37869 requested a full tablet report, sending...
I20260925 01:29:58.024677 3114 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:29:58.024806 3114 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:58.025324 3114 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
I20260925 01:29:58.025825 2596 ts_manager.cc:194] Registered new tserver with Master: 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2:36355)
I20260925 01:29:58.026253 1996 ts_manager.cc:194] Re-registered known tserver with Master: 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2:36355)
I20260925 01:29:58.027489 1996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:38793
I20260925 01:29:58.103408 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 2325
I20260925 01:29:58.119992 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:33863
--local_ip_for_outbound_sockets=127.27.160.3
--tserver_master_addrs=127.27.160.62:33983
--webserver_port=45097
--webserver_interface=127.27.160.3
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab
--principal=oryx/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869,127.27.160.60:41287 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:29:58.366132 2968 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
W20260925 01:29:58.459621 3126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:29:58.459973 3126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:29:58.460131 3126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:29:58.467659 3126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:29:58.467834 3126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:29:58.468008 3126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:29:58.482470 3126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab
--principal=oryx/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:33863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=45097
--tserver_master_addrs=127.27.160.62:33983,127.27.160.61:37869,127.27.160.60:41287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:29:58.484187 3126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:29:58.485683 3126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:29:58.498977 3134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:58.498937 3131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:29:58.499030 3132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:29:58.499958 3126 server_base.cc:1034] running on GCE node
I20260925 01:29:58.500566 3126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:29:58.501526 3126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:29:58.502753 3126 hybrid_clock.cc:648] HybridClock initialized: now 1790299798502702 us; error 43 us; skew 500 ppm
I20260925 01:29:58.503121 3126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:29:58 dist-test-slave-h6hf krb5kdc[1952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299798, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:29:58.506477 3126 init.cc:377] Logged in from keytab as oryx/127.27.160.3@KRBTEST.COM (short username oryx)
I20260925 01:29:58.507911 3126 webserver.cc:460] Webserver started at http://127.27.160.3:45097/ using document root <none> and password file <none>
I20260925 01:29:58.508558 3126 fs_manager.cc:359] Metadata directory not provided
I20260925 01:29:58.508680 3126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:29:58.514371 3126 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:29:58.517571 3141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:58.518929 3126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:29:58.519135 3126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "5b805c41709d45b1a9cd4d074a450adf"
format_stamp: "Formatted at 2026-09-25 01:29:55 on dist-test-slave-h6hf"
server_key: "982924fc466b89a07bc14d402440aaa6"
server_key_iv: "381876c70c7d013c096566b075f01818"
server_key_version: "encryptionkey@0"
I20260925 01:29:58.520147 3126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:29:58.538547 3126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:29:58.543124 3126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:29:58.543426 3126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:29:58.544428 3126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:29:58.545864 3126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:29:58.545989 3126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:58.546118 3126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:29:58.546206 3126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:29:58.580463 3126 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:33863
I20260925 01:29:58.580574 3253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:33863 every 8 connection(s)
I20260925 01:29:58.582073 3126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:29:58.591223 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 3126
Sep 25 01:29:58 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299798, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.61@KRBTEST.COM
Sep 25 01:29:58 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299798, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
Sep 25 01:29:58 dist-test-slave-h6hf krb5kdc[1952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299798, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.60@KRBTEST.COM
I20260925 01:29:58.611431 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.585137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:60339 (local address 127.27.160.62:33983)
0925 01:29:58.592922 (+ 7785us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.592928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.592947 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:29:58.593015 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.593020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.593104 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.594181 (+ 1077us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:58.594328 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.595352 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.597797 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.598446 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.598593 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.603211 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.603217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.603222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.603264 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.607238 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.608384 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.608392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.608398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.608594 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.609189 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.609197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.609203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.609709 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.609929 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.610767 (+ 838us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.611066 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7664,"thread_start_us":7558,"threads_started":1}
I20260925 01:29:58.612180 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.585913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.27.160.3:50729)
0925 01:29:58.586934 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:58.586991 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:58.590341 (+ 3350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:58.591192 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:58.591203 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:58.592251 (+ 1048us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:58.593326 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.593351 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.594877 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.594882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.596646 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.596658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.597678 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.597683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.597957 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.598785 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:58.598808 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:58.601668 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:58.606387 (+ 4719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.606398 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.606418 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.607089 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.607650 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.607654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.607658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.607932 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.608690 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:58.608701 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:58.609222 (+ 521us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.609564 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.610147 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":32,"spinlock_wait_cycles":70400,"thread_start_us":814,"threads_started":1}
I20260925 01:29:58.612357 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.586634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.3:60339)
0925 01:29:58.587328 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:58.587346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:58.589720 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:58.593311 (+ 3591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:58.593315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:58.593798 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:58.594032 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.594043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.595531 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.595536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.597605 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.597625 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.598666 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.598676 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.598947 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.599634 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:58.599664 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:58.602827 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:58.607437 (+ 4610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.607446 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.607452 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.608206 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.608749 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.608757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.608764 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.609012 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.610004 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:58.610010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:58.610222 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.610380 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.611856 (+ 1476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":35,"thread_start_us":1214,"threads_started":1}
I20260925 01:29:58.613870 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.585071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:50729 (local address 127.27.160.61:37869)
0925 01:29:58.585572 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.585576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.587167 (+ 1591us) server_negotiation.cc:407] Connection header received
0925 01:29:58.590509 (+ 3342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.590516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.590626 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.590721 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:58.593552 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.594684 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.596788 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.597309 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.597425 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.601936 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.601945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.601951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.601997 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.606211 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.607336 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.607341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.607344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.607498 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.608079 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.608085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.608092 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.608452 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.608608 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.611892 (+ 3284us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.612766 (+ 874us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260925 01:29:58.615240 3255 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:29:58.615649 3255 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:58.615654 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.588492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:54925 (local address 127.27.160.60:41287)
0925 01:29:58.590451 (+ 1959us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.590457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.590938 (+ 481us) server_negotiation.cc:407] Connection header received
0925 01:29:58.591166 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.591171 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.591246 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.591319 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:58.593615 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.595055 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.603138 (+ 8083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.604036 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.604179 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.608194 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.608200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.608205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.608243 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.611570 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.613011 (+ 1441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.613017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.613020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.613223 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.613685 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.613691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.613694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.614079 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.614215 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.614832 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.615155 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1840,"thread_start_us":1750,"threads_started":1}
I20260925 01:29:58.616853 3255 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
I20260925 01:29:58.617009 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.588399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:41287 (local address 127.27.160.3:54925)
0925 01:29:58.590853 (+ 2454us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:58.590870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:58.591027 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:58.591690 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:58.591696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:58.592413 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:29:58.593333 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.593358 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.595301 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.595309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.602910 (+ 7601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.602927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.604257 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.604263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.604446 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.605263 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:29:58.605284 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:29:58.607969 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:29:58.611736 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.611740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.611745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.612266 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.613371 (+ 1105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:29:58.613375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:29:58.613379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:29:58.613532 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:29:58.614297 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:29:58.614301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:29:58.614498 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.615898 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.616383 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2343,"mutex_wait_us":248,"thread_start_us":3764,"threads_started":1}
I20260925 01:29:58.618366 3254 heartbeater.cc:347] Connected to a master server at 127.27.160.60:41287
I20260925 01:29:58.618512 3254 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:58.618847 1996 ts_manager.cc:194] Re-registered known tserver with Master: 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863)
I20260925 01:29:58.618994 3254 heartbeater.cc:511] Master 127.27.160.60:41287 requested a full tablet report, sending...
I20260925 01:29:58.620275 1996 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:60339
I20260925 01:29:58.620417 2788 ts_manager.cc:194] Registered new tserver with Master: 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863)
I20260925 01:29:58.621616 3256 heartbeater.cc:347] Connected to a master server at 127.27.160.61:37869
I20260925 01:29:58.621755 3256 heartbeater.cc:464] Registering TS with master...
I20260925 01:29:58.622218 3256 heartbeater.cc:511] Master 127.27.160.61:37869 requested a full tablet report, sending...
I20260925 01:29:58.623268 2596 ts_manager.cc:194] Registered new tserver with Master: 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863)
I20260925 01:29:58.641018 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.625013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.27.160.60:41287)
0925 01:29:58.625214 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.625220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.625964 (+ 744us) server_negotiation.cc:407] Connection header received
0925 01:29:58.626139 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.626146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.626259 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.626369 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:58.628502 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.629325 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.631046 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.631677 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.631785 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.634800 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.634806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.634811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.634850 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.637928 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.638604 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.638612 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.638615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.638741 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.639188 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.639194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.639198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.639571 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.639812 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.640421 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.640659 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 01:29:58.642205 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.612357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.27.160.61:37869)
0925 01:29:58.614340 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.614345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.624924 (+ 10579us) server_negotiation.cc:407] Connection header received
0925 01:29:58.625219 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.625226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.625312 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.625384 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:58.627807 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.628922 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.630894 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.631521 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.631654 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.635255 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.635263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.635269 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.635314 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.638796 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.639633 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.639641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.639644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.639796 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.640230 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.640238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.640245 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.640608 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.640862 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.641572 (+ 710us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.641814 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1865,"thread_start_us":8521,"threads_started":1}
I20260925 01:29:58.642644 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.612766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.27.160.62:33983)
0925 01:29:58.613045 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.613052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.624135 (+ 11083us) server_negotiation.cc:407] Connection header received
0925 01:29:58.624331 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.624339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.624432 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.624548 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:58.627003 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.627707 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.629602 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.630218 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.630338 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.635515 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.635523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.635528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.635569 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.638945 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.639924 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.639929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.639931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.640054 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.640428 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.640433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.640436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.640725 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.640998 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.641655 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.641925 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 01:29:58.646466 1996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:29:58.653590 1996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:29:58.683935 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.674400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.27.160.3:33863)
0925 01:29:58.674983 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.674991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.676204 (+ 1213us) server_negotiation.cc:407] Connection header received
0925 01:29:58.676320 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.676327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.676532 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.676769 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:58.677408 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.678925 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.681373 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.682857 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.682985 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.683241 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.683410 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.683584 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":329,"threads_started":1}
I20260925 01:29:58.684297 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.674072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:33863 (local address 127.0.0.1:50624)
0925 01:29:58.674656 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:58.674684 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:58.676103 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:58.676733 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:58.676738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:58.676762 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:58.677218 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.677237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.679048 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.679055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.681231 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.681242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.683028 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.683034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.683216 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.683253 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.683366 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.683585 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":287,"threads_started":1}
I20260925 01:29:58.685738 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.674548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:36355 (local address 127.0.0.1:55610)
0925 01:29:58.676657 (+ 2109us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:58.677390 (+ 733us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:58.677627 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:58.678314 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:58.678321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:58.678354 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:58.678821 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.678835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.681359 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.681363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.682733 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.682744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.684909 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.684918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.685097 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.685152 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.685246 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.685370 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1949,"thread_start_us":1832,"threads_started":1}
I20260925 01:29:58.685726 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.674954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.27.160.2:36355)
0925 01:29:58.675577 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.675588 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.677485 (+ 1897us) server_negotiation.cc:407] Connection header received
0925 01:29:58.677818 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.677827 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.678117 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.678309 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:58.679005 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.680441 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.682877 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.684339 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.684474 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.684756 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.685288 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.685449 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":464,"threads_started":1}
I20260925 01:29:58.688159 3189 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab45223c895422a8fabb855075d21d8 (DEFAULT_TABLE table=test-table [id=42fcacc32ff34b40849f2696e50c7c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:58.688609 3048 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab45223c895422a8fabb855075d21d8 (DEFAULT_TABLE table=test-table [id=42fcacc32ff34b40849f2696e50c7c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:58.689514 3189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab45223c895422a8fabb855075d21d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:58.689951 3048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab45223c895422a8fabb855075d21d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:58.691784 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.677089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.27.160.1:40581)
0925 01:29:58.679773 (+ 2684us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.679781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.679808 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:29:58.679898 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.679906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.680136 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.680316 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:58.681204 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.682734 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.689172 (+ 6438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.690690 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.690827 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.691027 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.691313 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.691496 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2520,"thread_start_us":2376,"threads_started":1}
I20260925 01:29:58.691790 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.676720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40581 (local address 127.0.0.1:44848)
0925 01:29:58.677352 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:58.677370 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:58.677483 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:58.680333 (+ 2850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:58.680338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:58.680363 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:58.681012 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.681025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.682911 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.682919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.688956 (+ 6037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.688972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.690911 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.690918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.691106 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.691138 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.691233 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.691369 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":544,"threads_started":1}
I20260925 01:29:58.695636 2902 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab45223c895422a8fabb855075d21d8 (DEFAULT_TABLE table=test-table [id=42fcacc32ff34b40849f2696e50c7c38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:29:58.697002 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab45223c895422a8fabb855075d21d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:29:58.702548 3289 tablet_bootstrap.cc:492] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Bootstrap starting.
I20260925 01:29:58.706241 3289 tablet_bootstrap.cc:654] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:58.707299 3289 log.cc:824] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:58.710350 3289 tablet_bootstrap.cc:492] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: No bootstrap required, opened a new log
I20260925 01:29:58.710634 3289 ts_tablet_manager.cc:1397] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 01:29:58.712992 3292 tablet_bootstrap.cc:492] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Bootstrap starting.
I20260925 01:29:58.717067 3292 tablet_bootstrap.cc:654] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:58.717231 3290 tablet_bootstrap.cc:492] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: Bootstrap starting.
I20260925 01:29:58.718469 3292 log.cc:824] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:58.718562 3289 raft_consensus.cc:348] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.718990 3289 raft_consensus.cc:374] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:58.719125 3289 raft_consensus.cc:729] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b805c41709d45b1a9cd4d074a450adf, State: Initialized, Role: FOLLOWER
I20260925 01:29:58.719759 3289 consensus_queue.cc:260] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.721305 3290 tablet_bootstrap.cc:654] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: Neither blocks nor log segments found. Creating new log.
I20260925 01:29:58.721557 3289 ts_tablet_manager.cc:1428] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260925 01:29:58.721812 3255 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
I20260925 01:29:58.723448 3290 log.cc:824] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: Log is configured to *not* fsync() on all Append() calls
I20260925 01:29:58.727252 3292 tablet_bootstrap.cc:492] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: No bootstrap required, opened a new log
I20260925 01:29:58.727664 3292 ts_tablet_manager.cc:1397] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260925 01:29:58.730844 3290 tablet_bootstrap.cc:492] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: No bootstrap required, opened a new log
I20260925 01:29:58.731149 3290 ts_tablet_manager.cc:1397] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.005s
I20260925 01:29:58.738459 3292 raft_consensus.cc:348] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.738969 3292 raft_consensus.cc:374] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:58.739068 3292 raft_consensus.cc:729] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764191dcd7ec42758b4181b222acd471, State: Initialized, Role: FOLLOWER
I20260925 01:29:58.739749 3292 consensus_queue.cc:260] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.741791 3292 ts_tablet_manager.cc:1428] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 01:29:58.741699 3290 raft_consensus.cc:348] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.742131 3290 raft_consensus.cc:374] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:29:58.742223 3290 raft_consensus.cc:729] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c8f4062454451b8404bb1cf3f389fc, State: Initialized, Role: FOLLOWER
I20260925 01:29:58.742957 3290 consensus_queue.cc:260] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.745088 3296 raft_consensus.cc:482] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:29:58.745393 3296 raft_consensus.cc:504] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.746263 3114 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
I20260925 01:29:58.746891 3290 ts_tablet_manager.cc:1428] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260925 01:29:58.747308 3296 leader_election.cc:288] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863), 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2:36355)
I20260925 01:29:58.761662 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.748516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:47865 (local address 127.27.160.2:36355)
0925 01:29:58.748778 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.748787 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.754837 (+ 6050us) server_negotiation.cc:407] Connection header received
0925 01:29:58.755045 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.755050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.755113 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.755203 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:58.755745 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.756755 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.759339 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.760674 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.760852 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.761014 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.761238 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.761371 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 01:29:58.761793 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.747752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:36355 (local address 127.27.160.1:47865)
0925 01:29:58.754724 (+ 6972us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:58.754744 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:58.754908 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:58.755219 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:58.755223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:58.755246 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:58.755554 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.755585 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.756885 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.756890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.759188 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.759203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.760872 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.760876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.760987 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.761043 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.761211 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.761350 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6857,"mutex_wait_us":45,"thread_start_us":6533,"threads_started":1}
I20260925 01:29:58.761897 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.748397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:45421 (local address 127.27.160.3:33863)
0925 01:29:58.748602 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.748608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.755006 (+ 6398us) server_negotiation.cc:407] Connection header received
0925 01:29:58.755762 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.755768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.755873 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.755988 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:29:58.756501 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.757321 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.759637 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.760954 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.761189 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.761268 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.761507 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.761642 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:29:58.762259 3299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.748037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:33863 (local address 127.27.160.1:45421)
0925 01:29:58.754896 (+ 6859us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:58.754910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:29:58.755504 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:29:58.756022 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:29:58.756030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:29:58.756054 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:29:58.756340 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.756350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.757482 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.757490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.759470 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:29:58.759479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.761191 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.761199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:29:58.761350 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.761379 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:29:58.761468 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 01:29:58.761603 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6740,"mutex_wait_us":1,"thread_start_us":6590,"threads_started":1}
I20260925 01:29:58.762980 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab45223c895422a8fabb855075d21d8" candidate_uuid: "764191dcd7ec42758b4181b222acd471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c8f4062454451b8404bb1cf3f389fc" is_pre_election: true
I20260925 01:29:58.763305 3209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab45223c895422a8fabb855075d21d8" candidate_uuid: "764191dcd7ec42758b4181b222acd471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b805c41709d45b1a9cd4d074a450adf" is_pre_election: true
I20260925 01:29:58.763518 3068 raft_consensus.cc:2454] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 764191dcd7ec42758b4181b222acd471 in term 0.
I20260925 01:29:58.763821 3209 raft_consensus.cc:2454] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 764191dcd7ec42758b4181b222acd471 in term 0.
I20260925 01:29:58.764321 2855 leader_election.cc:302] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22c8f4062454451b8404bb1cf3f389fc, 764191dcd7ec42758b4181b222acd471; no voters:
I20260925 01:29:58.764731 3296 raft_consensus.cc:2781] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:29:58.764886 3296 raft_consensus.cc:482] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:29:58.764989 3296 raft_consensus.cc:3037] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:58.767292 3296 raft_consensus.cc:504] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.767885 3296 leader_election.cc:288] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [CANDIDATE]: Term 1 election: Requested vote from peers 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863), 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2:36355)
I20260925 01:29:58.768262 3209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab45223c895422a8fabb855075d21d8" candidate_uuid: "764191dcd7ec42758b4181b222acd471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b805c41709d45b1a9cd4d074a450adf"
I20260925 01:29:58.768299 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab45223c895422a8fabb855075d21d8" candidate_uuid: "764191dcd7ec42758b4181b222acd471" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22c8f4062454451b8404bb1cf3f389fc"
I20260925 01:29:58.768435 3209 raft_consensus.cc:3037] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:58.768522 3068 raft_consensus.cc:3037] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:29:58.770684 3209 raft_consensus.cc:2454] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 764191dcd7ec42758b4181b222acd471 in term 1.
I20260925 01:29:58.770809 3068 raft_consensus.cc:2454] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 764191dcd7ec42758b4181b222acd471 in term 1.
I20260925 01:29:58.771085 2857 leader_election.cc:302] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b805c41709d45b1a9cd4d074a450adf, 764191dcd7ec42758b4181b222acd471; no voters:
I20260925 01:29:58.771375 3296 raft_consensus.cc:2781] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:29:58.771811 3296 raft_consensus.cc:686] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 1 LEADER]: Becoming Leader. State: Replica: 764191dcd7ec42758b4181b222acd471, State: Running, Role: LEADER
I20260925 01:29:58.772363 3296 consensus_queue.cc:237] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:29:58.777779 1997 catalog_manager.cc:5495] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 reported cstate change: term changed from 0 to 1, leader changed from <none> to 764191dcd7ec42758b4181b222acd471 (127.27.160.1). New cstate: current_term: 1 leader_uuid: "764191dcd7ec42758b4181b222acd471" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } health_report { overall_health: HEALTHY } } }
I20260925 01:29:58.812906 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.807423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.27.160.1:40581)
0925 01:29:58.807625 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.807632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.807738 (+ 106us) server_negotiation.cc:407] Connection header received
0925 01:29:58.807982 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.807986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.808090 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.808242 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:29:58.808610 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.809403 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.810758 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.811164 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.811298 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.811779 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:29:58.812350 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:29:58.812507 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.812567 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.812648 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 01:29:58.819085 2970 tablet.cc:2369] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:58.821866 3068 raft_consensus.cc:1261] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 1 FOLLOWER]: Refusing update from remote peer 764191dcd7ec42758b4181b222acd471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:58.821864 3209 raft_consensus.cc:1261] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 1 FOLLOWER]: Refusing update from remote peer 764191dcd7ec42758b4181b222acd471: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:29:58.822926 3300 consensus_queue.cc:1035] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:29:58.823323 3296 consensus_queue.cc:1035] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:29:58.835132 3257 tablet.cc:2369] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:29:58.839968 3304 mvcc.cc:205] Tried to move back new op lower bound from 7333067975963947008 to 7333067975775711232. Current Snapshot: MvccSnapshot[applied={T|T < 7333067975963947008}]
I20260925 01:29:58.865769 1997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:58.866068 1997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:29:58.873733 1997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e268b00cbb64f1fa59b486526de6a63: Sending DeleteTablet for 3 replicas of tablet 5ab45223c895422a8fabb855075d21d8
I20260925 01:29:58.874917 3189 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab45223c895422a8fabb855075d21d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:58 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:50624
I20260925 01:29:58.875036 2902 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab45223c895422a8fabb855075d21d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:58 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:44848
I20260925 01:29:58.875754 3048 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab45223c895422a8fabb855075d21d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:29:58 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:55610
I20260925 01:29:58.875854 3313 tablet_replica.cc:330] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: stopping tablet replica
I20260925 01:29:58.876408 3313 raft_consensus.cc:2229] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:58.877018 3313 raft_consensus.cc:2258] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:58.879401 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 1965
I20260925 01:29:58.880497 2027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:29:58.883679 3313 ts_tablet_manager.cc:1905] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:58.885147 3314 tablet_replica.cc:330] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: stopping tablet replica
I20260925 01:29:58.885676 3314 raft_consensus.cc:2229] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:29:58.886507 3314 raft_consensus.cc:2258] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:58.891350 3313 ts_tablet_manager.cc:1918] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:58.891794 3313 log.cc:1196] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/5ab45223c895422a8fabb855075d21d8
I20260925 01:29:58.892432 3313 ts_tablet_manager.cc:1939] T 5ab45223c895422a8fabb855075d21d8 P 5b805c41709d45b1a9cd4d074a450adf: Deleting consensus metadata
I20260925 01:29:58.896812 3314 ts_tablet_manager.cc:1905] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:58.898852 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.878004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.27.160.61:37869)
0925 01:29:58.878202 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.878208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.878224 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:29:58.878275 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.878279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.878332 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.878436 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:58.881478 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.882510 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.884680 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.885359 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.885461 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.889554 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.889561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.889568 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.889617 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.894218 (+ 4601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.895294 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.895303 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.895308 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.895533 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.896269 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.896278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.896284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.896786 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.897221 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.897982 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.898292 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:29:58.899313 3315 tablet_replica.cc:330] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: stopping tablet replica
I20260925 01:29:58.899782 3315 raft_consensus.cc:2229] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:29:58.900343 3315 raft_consensus.cc:2258] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:29:58.903319 3315 ts_tablet_manager.cc:1905] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:29:58.907830 3314 ts_tablet_manager.cc:1918] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:58.908032 3314 log.cc:1196] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/5ab45223c895422a8fabb855075d21d8
I20260925 01:29:58.908725 3314 ts_tablet_manager.cc:1939] T 5ab45223c895422a8fabb855075d21d8 P 764191dcd7ec42758b4181b222acd471: Deleting consensus metadata
I20260925 01:29:58.916352 3315 ts_tablet_manager.cc:1918] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:29:58.916522 3315 log.cc:1196] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/5ab45223c895422a8fabb855075d21d8
I20260925 01:29:58.917140 3315 ts_tablet_manager.cc:1939] T 5ab45223c895422a8fabb855075d21d8 P 22c8f4062454451b8404bb1cf3f389fc: Deleting consensus metadata
I20260925 01:29:58.918176 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:58.878712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.27.160.60:41287)
0925 01:29:58.878874 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:58.878881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:58.901463 (+ 22582us) server_negotiation.cc:407] Connection header received
0925 01:29:58.901540 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:58.901545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:58.901670 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:58.901762 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:58.903193 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.904023 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.905931 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:58.906594 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:58.906708 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:58.910866 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:58.910874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:58.910879 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:58.910923 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:58.914556 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.915476 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.915482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.915488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.915639 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:58.916083 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:58.916088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:58.916091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:58.916541 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:58.916868 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:58.917558 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 01:29:58.917799 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:29:59.027853 1983 catalog_manager.cc:4841] TS 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863): tablet 5ab45223c895422a8fabb855075d21d8 (table test-table [id=42fcacc32ff34b40849f2696e50c7c38]) successfully deleted
I20260925 01:29:59.027980 1981 catalog_manager.cc:4841] TS 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2:36355): tablet 5ab45223c895422a8fabb855075d21d8 (table test-table [id=42fcacc32ff34b40849f2696e50c7c38]) successfully deleted
I20260925 01:29:59.028376 1981 catalog_manager.cc:4841] TS 764191dcd7ec42758b4181b222acd471 (127.27.160.1:40581): tablet 5ab45223c895422a8fabb855075d21d8 (table test-table [id=42fcacc32ff34b40849f2696e50c7c38]) successfully deleted
I20260925 01:29:59.028648 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 1965
I20260925 01:29:59.073160 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:59.054901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.27.160.61:37869)
0925 01:29:59.055093 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:59.055101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:59.055727 (+ 626us) server_negotiation.cc:407] Connection header received
0925 01:29:59.056058 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:59.056065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:59.056147 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:59.056224 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:59.059316 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:59.060452 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:59.062483 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:59.063187 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:59.063307 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:59.067097 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:59.067106 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:59.067113 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:59.067161 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:59.070416 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:59.071062 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:59.071068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:59.071072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:59.071203 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:59.071581 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:59.071586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:59.071594 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:59.071899 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:59.072170 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:59.072514 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 01:29:59.072755 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:29:59.075274 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:59.056331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.27.160.60:41287)
0925 01:29:59.056511 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:29:59.056518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:29:59.056781 (+ 263us) server_negotiation.cc:407] Connection header received
0925 01:29:59.057080 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:29:59.057088 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:29:59.057271 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:29:59.057361 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:29:59.061061 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:59.062621 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:59.064789 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:29:59.065413 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:29:59.065544 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:29:59.068382 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:29:59.068389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:29:59.068394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:29:59.068436 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:29:59.071789 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:59.072604 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:59.072610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:59.072613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:59.072767 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:29:59.073227 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:29:59.073233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:29:59.073235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:29:59.073625 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:29:59.073890 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 01:29:59.074708 (+ 818us) server_negotiation.cc:299] Negotiation successful
0925 01:29:59.074944 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:29:59.848474 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:59.847489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.3:53711)
0925 01:29:59.848023 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:59.848175 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":296,"threads_started":1}
W20260925 01:29:59.848904 3255 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:33983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
I20260925 01:29:59.855669 3326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:59.854841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.2:43811)
0925 01:29:59.855318 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:59.855448 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
W20260925 01:29:59.856176 3114 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:33983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
I20260925 01:29:59.863300 3327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:29:59.862573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.1:50529)
0925 01:29:59.862983 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 01:29:59.863142 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":205,"threads_started":1}
W20260925 01:29:59.863698 2968 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:33983 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
I20260925 01:30:00.373208 3328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e268b00cbb64f1fa59b486526de6a63)
I20260925 01:30:00.373543 3328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } }
I20260925 01:30:00.373788 3329 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5e268b00cbb64f1fa59b486526de6a63)
I20260925 01:30:00.373957 3329 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } }
I20260925 01:30:00.374858 3329 leader_election.cc:288] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e268b00cbb64f1fa59b486526de6a63 (127.27.160.62:33983), 4017e314eba74525b086cc1dcbd5e905 (127.27.160.61:37869)
I20260925 01:30:00.375247 3328 leader_election.cc:288] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e268b00cbb64f1fa59b486526de6a63 (127.27.160.62:33983), c8dcccfd51814f85b39ff6635a86ff4f (127.27.160.60:41287)
I20260925 01:30:00.376503 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.374978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42192)
0925 01:30:00.375854 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.376133 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":670,"threads_started":1}
W20260925 01:30:00.376845 2773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
I20260925 01:30:00.378386 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.375411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42194)
0925 01:30:00.377799 (+ 2388us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.378047 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2212,"thread_start_us":2012,"threads_started":1}
I20260925 01:30:00.379318 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.378642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42204)
0925 01:30:00.378948 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.379144 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
W20260925 01:30:00.379253 2581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
W20260925 01:30:00.379556 2773 leader_election.cc:334] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e268b00cbb64f1fa59b486526de6a63 (127.27.160.62:33983): Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
I20260925 01:30:00.381098 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.380569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42208)
0925 01:30:00.380730 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.380906 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:30:00.381330 2581 leader_election.cc:334] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5e268b00cbb64f1fa59b486526de6a63 (127.27.160.62:33983): Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
I20260925 01:30:00.386170 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.375238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.27.160.61:37869)
0925 01:30:00.378567 (+ 3329us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.378576 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.378599 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:30:00.378682 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.378690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.378787 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.379450 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:00.379745 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.381148 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.383690 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.385152 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.385298 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.385389 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.385714 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.385834 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":3156,"thread_start_us":2950,"threads_started":1}
I20260925 01:30:00.386197 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.375035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.0.0.1:38212)
0925 01:30:00.376734 (+ 1699us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.376758 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:00.376942 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:00.378987 (+ 2045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:00.378993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:00.379020 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:00.379295 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.379312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.381399 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.381404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.383480 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.383528 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.385339 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.385345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.385538 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.385573 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:30:00.385667 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:30:00.385846 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1590,"thread_start_us":2199,"threads_started":1}
I20260925 01:30:00.386819 2606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4017e314eba74525b086cc1dcbd5e905" is_pre_election: true
I20260925 01:30:00.387147 2606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8dcccfd51814f85b39ff6635a86ff4f in term 1.
I20260925 01:30:00.387656 3333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.377831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:41287 (local address 127.0.0.1:57050)
0925 01:30:00.378578 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.378594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:00.378710 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:00.380373 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:00.380378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:00.380726 (+ 348us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:00.381147 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.381157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.383034 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.383037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.385177 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.385189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.386884 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.386888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.387077 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.387113 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:30:00.387192 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 01:30:00.387334 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":47,"thread_start_us":1142,"threads_started":1}
I20260925 01:30:00.387629 2774 leader_election.cc:302] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4017e314eba74525b086cc1dcbd5e905, c8dcccfd51814f85b39ff6635a86ff4f; no voters: 5e268b00cbb64f1fa59b486526de6a63
I20260925 01:30:00.387748 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.378283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.27.160.60:41287)
0925 01:30:00.380065 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.380071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.380096 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:30:00.380152 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.380157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.380231 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.380337 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:00.381655 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.382879 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.385314 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.386692 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.386824 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.386924 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.387235 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.387360 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1636,"thread_start_us":1333,"threads_started":1}
I20260925 01:30:00.388294 3329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 01:30:00.388427 3329 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5e268b00cbb64f1fa59b486526de6a63)
I20260925 01:30:00.388576 3329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:30:00.388403 2798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4017e314eba74525b086cc1dcbd5e905" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" is_pre_election: true
I20260925 01:30:00.390739 3329 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } }
I20260925 01:30:00.391214 2798 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4017e314eba74525b086cc1dcbd5e905 in current term 2: Already voted for candidate c8dcccfd51814f85b39ff6635a86ff4f in this term.
I20260925 01:30:00.391492 3329 leader_election.cc:288] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [CANDIDATE]: Term 2 election: Requested vote from peers 5e268b00cbb64f1fa59b486526de6a63 (127.27.160.62:33983), 4017e314eba74525b086cc1dcbd5e905 (127.27.160.61:37869)
I20260925 01:30:00.392061 2606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4017e314eba74525b086cc1dcbd5e905"
I20260925 01:30:00.392298 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.391580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42220)
0925 01:30:00.391999 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.392080 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306}
I20260925 01:30:00.392306 2606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:30:00.392786 2581 leader_election.cc:302] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4017e314eba74525b086cc1dcbd5e905; no voters: 5e268b00cbb64f1fa59b486526de6a63, c8dcccfd51814f85b39ff6635a86ff4f
I20260925 01:30:00.393332 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.392889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42228)
0925 01:30:00.393102 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.393241 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260925 01:30:00.393517 2773 leader_election.cc:334] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5e268b00cbb64f1fa59b486526de6a63 (127.27.160.62:33983): Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
I20260925 01:30:00.394798 2606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8dcccfd51814f85b39ff6635a86ff4f in term 2.
I20260925 01:30:00.395030 3328 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:30:00.395141 2774 leader_election.cc:302] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4017e314eba74525b086cc1dcbd5e905, c8dcccfd51814f85b39ff6635a86ff4f; no voters: 5e268b00cbb64f1fa59b486526de6a63
I20260925 01:30:00.395427 3329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:30:00.395828 3329 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [term 2 LEADER]: Becoming Leader. State: Replica: c8dcccfd51814f85b39ff6635a86ff4f, State: Running, Role: LEADER
I20260925 01:30:00.396200 3329 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } }
I20260925 01:30:00.397993 3338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8dcccfd51814f85b39ff6635a86ff4f. Latest consensus state: current_term: 2 leader_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:30:00.398293 3338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:00.398865 3340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:30:00.402426 3340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:30:00.403223 3340 catalog_manager.cc:1259] Loaded cluster ID: f4ff7f0d1de44a2d928d3ed9de76f417
I20260925 01:30:00.403321 3340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:30:00.404515 3340 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:30:00.405671 3340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Loaded TSK: 0
I20260925 01:30:00.406528 3340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:30:00.462518 2788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:30:00.469483 2788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:30:00.473646 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.473242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42250)
0925 01:30:00.473417 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.473493 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 01:30:00.473702 2606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [term 2 FOLLOWER]: Refusing update from remote peer c8dcccfd51814f85b39ff6635a86ff4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 01:30:00.474514 3338 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 01:30:00.474713 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.474281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42266)
0925 01:30:00.474447 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.474545 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 01:30:00.474880 2773 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f -> Peer 5e268b00cbb64f1fa59b486526de6a63 (127.27.160.62:33983): Couldn't send request to peer 5e268b00cbb64f1fa59b486526de6a63. Status: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 01:30:00.477438 3328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8dcccfd51814f85b39ff6635a86ff4f. Latest consensus state: current_term: 2 leader_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:30:00.477674 3328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:30:00.479008 3338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:30:00.479336 3338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:00.479717 3328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5e268b00cbb64f1fa59b486526de6a63" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } peers { permanent_uuid: "4017e314eba74525b086cc1dcbd5e905" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 37869 } attrs { promote: false } } peers { permanent_uuid: "c8dcccfd51814f85b39ff6635a86ff4f" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 41287 } attrs { promote: false } } }
I20260925 01:30:00.480011 3328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4017e314eba74525b086cc1dcbd5e905 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:30:00.497349 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.487751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:33863 (local address 127.0.0.1:52156)
0925 01:30:00.487922 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.487942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:00.488050 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:00.489284 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:00.489289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:00.489315 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:00.489571 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.489587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.491659 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.491668 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.494224 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.494237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.496376 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.496385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.496675 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.496736 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 01:30:00.496817 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 01:30:00.496970 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 01:30:00.499140 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.488296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.27.160.3:33863)
0925 01:30:00.488874 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.488884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.488910 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:30:00.489001 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.489007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.489104 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.489748 (+ 644us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:00.489984 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.491449 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.494489 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.496160 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.498268 (+ 2108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.498431 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.498496 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.498707 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":459,"threads_started":1}
I20260925 01:30:00.500269 3189 tablet_service.cc:1467] Processing CreateTablet for tablet 2a49847c982144209db4cc1f101bbd1d (DEFAULT_TABLE table=test-table [id=50b74a6535b646d2b0695e76982cf644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:00.500876 3189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a49847c982144209db4cc1f101bbd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:00.507808 3351 tablet_bootstrap.cc:492] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: Bootstrap starting.
I20260925 01:30:00.507932 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.487979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:36355 (local address 127.0.0.1:33492)
0925 01:30:00.488157 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.488177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:00.488262 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:00.499095 (+ 10833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:00.499103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:00.499138 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:00.499543 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.499554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.503307 (+ 3753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.503313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.505422 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.505437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.507157 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.507167 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.507366 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.507399 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:30:00.507469 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 01:30:00.507620 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 01:30:00.508836 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.488633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.27.160.2:36355)
0925 01:30:00.497976 (+ 9343us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.497985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.498010 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:30:00.498106 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.498115 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.498216 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.499712 (+ 1496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:00.499908 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.501331 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.505637 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.506844 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.508253 (+ 1409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.508360 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.508410 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.508559 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9190,"thread_start_us":9090,"threads_started":1}
I20260925 01:30:00.509554 3048 tablet_service.cc:1467] Processing CreateTablet for tablet 2a49847c982144209db4cc1f101bbd1d (DEFAULT_TABLE table=test-table [id=50b74a6535b646d2b0695e76982cf644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:00.509945 3351 tablet_bootstrap.cc:654] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:00.509981 3048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a49847c982144209db4cc1f101bbd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:00.514901 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.488368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40581 (local address 127.0.0.1:35796)
0925 01:30:00.497638 (+ 9270us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.497661 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:00.497801 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:00.508356 (+ 10555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:00.508360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:00.508390 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:00.508560 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.508570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.510002 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.510020 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.511893 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.511911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.514090 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.514096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.514316 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.514348 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:30:00.514450 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 01:30:00.514592 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9142,"thread_start_us":13648,"threads_started":1}
I20260925 01:30:00.515947 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.488516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.27.160.1:40581)
0925 01:30:00.508023 (+ 19507us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.508030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.508064 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:30:00.508137 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.508142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.508231 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.508337 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:00.508719 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.509823 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.512108 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.513890 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.514070 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.514212 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.515406 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.515554 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19376,"thread_start_us":19928,"threads_started":1}
I20260925 01:30:00.516299 3353 tablet_bootstrap.cc:492] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: Bootstrap starting.
I20260925 01:30:00.517088 2902 tablet_service.cc:1467] Processing CreateTablet for tablet 2a49847c982144209db4cc1f101bbd1d (DEFAULT_TABLE table=test-table [id=50b74a6535b646d2b0695e76982cf644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:00.517594 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a49847c982144209db4cc1f101bbd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:00.518587 3351 tablet_bootstrap.cc:492] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: No bootstrap required, opened a new log
I20260925 01:30:00.518707 3353 tablet_bootstrap.cc:654] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:00.518764 3351 ts_tablet_manager.cc:1397] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 01:30:00.519652 3351 raft_consensus.cc:348] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.519886 3351 raft_consensus.cc:374] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:00.519994 3351 raft_consensus.cc:729] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b805c41709d45b1a9cd4d074a450adf, State: Initialized, Role: FOLLOWER
I20260925 01:30:00.520262 3351 consensus_queue.cc:260] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.521160 3351 ts_tablet_manager.cc:1428] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 01:30:00.521602 3353 tablet_bootstrap.cc:492] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: No bootstrap required, opened a new log
I20260925 01:30:00.521754 3353 ts_tablet_manager.cc:1397] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 01:30:00.522377 3353 raft_consensus.cc:348] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.522544 3353 raft_consensus.cc:374] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:00.522665 3353 raft_consensus.cc:729] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22c8f4062454451b8404bb1cf3f389fc, State: Initialized, Role: FOLLOWER
I20260925 01:30:00.522938 3353 consensus_queue.cc:260] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.523283 3356 tablet_bootstrap.cc:492] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: Bootstrap starting.
I20260925 01:30:00.525415 3356 tablet_bootstrap.cc:654] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:00.526129 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.525022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.3:56835)
0925 01:30:00.525722 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.525877 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"threads_started":1}
I20260925 01:30:00.527842 3356 tablet_bootstrap.cc:492] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: No bootstrap required, opened a new log
I20260925 01:30:00.527981 3356 ts_tablet_manager.cc:1397] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 01:30:00.528756 3356 raft_consensus.cc:348] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.529003 3356 raft_consensus.cc:374] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:00.529111 3356 raft_consensus.cc:729] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764191dcd7ec42758b4181b222acd471, State: Initialized, Role: FOLLOWER
I20260925 01:30:00.529458 3356 consensus_queue.cc:260] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.530068 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.527846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.3:58981)
0925 01:30:00.529742 (+ 1896us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.529841 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1780}
I20260925 01:30:00.530182 3254 heartbeater.cc:503] Master 127.27.160.60:41287 was elected leader, sending a full tablet report...
I20260925 01:30:00.531046 3356 ts_tablet_manager.cc:1428] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:00.532392 3353 ts_tablet_manager.cc:1428] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260925 01:30:00.534310 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.533306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.2:46921)
0925 01:30:00.533853 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.534046 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"threads_started":1}
I20260925 01:30:00.534653 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.531637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.1:44985)
0925 01:30:00.534287 (+ 2650us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.534420 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2530,"threads_started":1}
I20260925 01:30:00.535653 2967 heartbeater.cc:503] Master 127.27.160.60:41287 was elected leader, sending a full tablet report...
I20260925 01:30:00.536367 3113 heartbeater.cc:503] Master 127.27.160.60:41287 was elected leader, sending a full tablet report...
I20260925 01:30:00.536580 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.536200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.1:46509)
0925 01:30:00.536329 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.536418 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 01:30:00.542622 3357 raft_consensus.cc:482] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:30:00.542861 3357 raft_consensus.cc:504] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.544385 3357 leader_election.cc:288] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863), 764191dcd7ec42758b4181b222acd471 (127.27.160.1:40581)
I20260925 01:30:00.550969 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.544892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:35457 (local address 127.27.160.3:33863)
0925 01:30:00.545322 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.545328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.545349 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:30:00.545444 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.545452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.545531 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.545627 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:00.546194 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.547043 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.548819 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.549982 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.550127 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.550215 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.550575 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.550715 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260925 01:30:00.550997 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.544777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:39597 (local address 127.27.160.1:40581)
0925 01:30:00.545165 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.545174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.545507 (+ 333us) server_negotiation.cc:407] Connection header received
0925 01:30:00.545689 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.545694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.545772 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.545866 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:00.546485 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.547198 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.549007 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.550031 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.550219 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.550294 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.550643 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.550774 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260925 01:30:00.551000 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.544665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:33863 (local address 127.27.160.2:35457)
0925 01:30:00.544933 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.544959 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:00.545304 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:00.545664 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:00.545670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:00.545698 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:00.546036 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.546046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.547167 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.547172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.548650 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.548665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.550163 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.550171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.550411 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.550453 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 01:30:00.550528 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 01:30:00.550683 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":6}
I20260925 01:30:00.551179 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.544710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40581 (local address 127.27.160.2:39597)
0925 01:30:00.545393 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.545410 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:00.545545 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:00.545904 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:00.545907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:00.545927 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:00.546337 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.546348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.547355 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.547366 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.548857 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:00.548871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.550311 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.550318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:00.550461 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.550483 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 01:30:00.550593 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 01:30:00.550715 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":26,"thread_start_us":410,"threads_started":1}
I20260925 01:30:00.552067 3209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a49847c982144209db4cc1f101bbd1d" candidate_uuid: "22c8f4062454451b8404bb1cf3f389fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b805c41709d45b1a9cd4d074a450adf" is_pre_election: true
I20260925 01:30:00.552014 2922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a49847c982144209db4cc1f101bbd1d" candidate_uuid: "22c8f4062454451b8404bb1cf3f389fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "764191dcd7ec42758b4181b222acd471" is_pre_election: true
I20260925 01:30:00.552306 3209 raft_consensus.cc:2454] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22c8f4062454451b8404bb1cf3f389fc in term 0.
I20260925 01:30:00.552412 2922 raft_consensus.cc:2454] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22c8f4062454451b8404bb1cf3f389fc in term 0.
I20260925 01:30:00.552917 3003 leader_election.cc:302] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22c8f4062454451b8404bb1cf3f389fc, 5b805c41709d45b1a9cd4d074a450adf, 764191dcd7ec42758b4181b222acd471; no voters:
I20260925 01:30:00.553364 3357 raft_consensus.cc:2781] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:30:00.553489 3357 raft_consensus.cc:482] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:30:00.553630 3357 raft_consensus.cc:3037] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:00.555701 3357 raft_consensus.cc:504] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.556386 3357 leader_election.cc:288] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [CANDIDATE]: Term 1 election: Requested vote from peers 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863), 764191dcd7ec42758b4181b222acd471 (127.27.160.1:40581)
I20260925 01:30:00.556789 3209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a49847c982144209db4cc1f101bbd1d" candidate_uuid: "22c8f4062454451b8404bb1cf3f389fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b805c41709d45b1a9cd4d074a450adf"
I20260925 01:30:00.556842 2922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a49847c982144209db4cc1f101bbd1d" candidate_uuid: "22c8f4062454451b8404bb1cf3f389fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "764191dcd7ec42758b4181b222acd471"
I20260925 01:30:00.556993 3209 raft_consensus.cc:3037] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:00.557021 2922 raft_consensus.cc:3037] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:00.559178 3209 raft_consensus.cc:2454] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22c8f4062454451b8404bb1cf3f389fc in term 1.
I20260925 01:30:00.559552 2922 raft_consensus.cc:2454] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22c8f4062454451b8404bb1cf3f389fc in term 1.
I20260925 01:30:00.559638 3003 leader_election.cc:302] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22c8f4062454451b8404bb1cf3f389fc, 5b805c41709d45b1a9cd4d074a450adf; no voters:
I20260925 01:30:00.559985 3357 raft_consensus.cc:2781] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:00.560530 3357 raft_consensus.cc:686] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 1 LEADER]: Becoming Leader. State: Replica: 22c8f4062454451b8404bb1cf3f389fc, State: Running, Role: LEADER
I20260925 01:30:00.560962 3357 consensus_queue.cc:237] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } }
I20260925 01:30:00.561682 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.561314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.2:52249)
0925 01:30:00.561466 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.561555 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 01:30:00.564492 2788 catalog_manager.cc:5495] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2). New cstate: current_term: 1 leader_uuid: "22c8f4062454451b8404bb1cf3f389fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22c8f4062454451b8404bb1cf3f389fc" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 } health_report { overall_health: UNKNOWN } } }
I20260925 01:30:00.586948 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.580312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.27.160.2:36355)
0925 01:30:00.580504 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.580509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.580579 (+ 70us) server_negotiation.cc:407] Connection header received
0925 01:30:00.580846 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.580851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.580944 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.581078 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:00.581499 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.582346 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.584320 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.584985 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.585170 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.585631 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:00.586357 (+ 726us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:00.586474 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.586604 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.586677 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:30:00.591289 3209 raft_consensus.cc:1261] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 1 FOLLOWER]: Refusing update from remote peer 22c8f4062454451b8404bb1cf3f389fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:00.591657 2922 raft_consensus.cc:1261] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 1 FOLLOWER]: Refusing update from remote peer 22c8f4062454451b8404bb1cf3f389fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:00.592006 3364 consensus_queue.cc:1035] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b805c41709d45b1a9cd4d074a450adf" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:00.592382 3357 consensus_queue.cc:1035] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "764191dcd7ec42758b4181b222acd471" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:00.595090 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.594535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.1:36741)
0925 01:30:00.594755 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.594874 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 01:30:00.595289 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.594765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.3:59259)
0925 01:30:00.595020 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.595119 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260925 01:30:00.602423 3369 mvcc.cc:205] Tried to move back new op lower bound from 7333067983215239168 to 7333067983101968384. Current Snapshot: MvccSnapshot[applied={T|T < 7333067983215239168}]
I20260925 01:30:00.603681 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.603202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.1:49947)
0925 01:30:00.603438 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.603534 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260925 01:30:00.604029 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.603626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.2:40265)
0925 01:30:00.603764 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.603867 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 01:30:00.605007 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.604151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.3:47541)
0925 01:30:00.604748 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.604838 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
I20260925 01:30:00.605295 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.604784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.2:40429)
0925 01:30:00.605038 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:00.605151 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 01:30:00.605662 3368 mvcc.cc:205] Tried to move back new op lower bound from 7333067983215239168 to 7333067983101968384. Current Snapshot: MvccSnapshot[applied={T|T < 7333067983215239168}]
I20260925 01:30:00.618712 2786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:00.618963 2786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:00.624837 2786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f: Sending DeleteTablet for 3 replicas of tablet 2a49847c982144209db4cc1f101bbd1d
I20260925 01:30:00.625949 3189 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a49847c982144209db4cc1f101bbd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:00 UTC) from {username='oryx', principal='oryx/127.27.160.60@KRBTEST.COM'} at 127.0.0.1:52156
I20260925 01:30:00.626139 2902 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a49847c982144209db4cc1f101bbd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:00 UTC) from {username='oryx', principal='oryx/127.27.160.60@KRBTEST.COM'} at 127.0.0.1:35796
I20260925 01:30:00.626874 3048 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a49847c982144209db4cc1f101bbd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:00 UTC) from {username='oryx', principal='oryx/127.27.160.60@KRBTEST.COM'} at 127.0.0.1:33492
I20260925 01:30:00.627779 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 2769
I20260925 01:30:00.630862 3379 tablet_replica.cc:330] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: stopping tablet replica
I20260925 01:30:00.631245 3379 raft_consensus.cc:2229] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:30:00.632028 3379 raft_consensus.cc:2258] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:00.632627 3378 tablet_replica.cc:330] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: stopping tablet replica
I20260925 01:30:00.632907 3378 raft_consensus.cc:2229] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:00.633433 3378 raft_consensus.cc:2258] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:00.634754 3379 ts_tablet_manager.cc:1905] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:00.636117 3378 ts_tablet_manager.cc:1905] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:00.643803 3379 ts_tablet_manager.cc:1918] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:00.650671 3377 tablet_replica.cc:330] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: stopping tablet replica
I20260925 01:30:00.650898 3377 raft_consensus.cc:2229] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:00.651063 3378 ts_tablet_manager.cc:1918] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:00.651230 3378 log.cc:1196] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/2a49847c982144209db4cc1f101bbd1d
I20260925 01:30:00.651229 3377 raft_consensus.cc:2258] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:00.651768 3378 ts_tablet_manager.cc:1939] T 2a49847c982144209db4cc1f101bbd1d P 764191dcd7ec42758b4181b222acd471: Deleting consensus metadata
I20260925 01:30:00.652997 3379 log.cc:1196] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/2a49847c982144209db4cc1f101bbd1d
I20260925 01:30:00.653266 2772 catalog_manager.cc:4841] TS 764191dcd7ec42758b4181b222acd471 (127.27.160.1:40581): tablet 2a49847c982144209db4cc1f101bbd1d (table test-table [id=50b74a6535b646d2b0695e76982cf644]) successfully deleted
I20260925 01:30:00.653611 3379 ts_tablet_manager.cc:1939] T 2a49847c982144209db4cc1f101bbd1d P 22c8f4062454451b8404bb1cf3f389fc: Deleting consensus metadata
I20260925 01:30:00.654557 3377 ts_tablet_manager.cc:1905] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:00.654913 2772 catalog_manager.cc:4841] TS 22c8f4062454451b8404bb1cf3f389fc (127.27.160.2:36355): tablet 2a49847c982144209db4cc1f101bbd1d (table test-table [id=50b74a6535b646d2b0695e76982cf644]) successfully deleted
I20260925 01:30:00.661795 3377 ts_tablet_manager.cc:1918] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:00.661939 3377 log.cc:1196] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/2a49847c982144209db4cc1f101bbd1d
I20260925 01:30:00.662231 3349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.629055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.27.160.1:40581)
0925 01:30:00.629301 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.629311 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.640040 (+ 10729us) server_negotiation.cc:407] Connection header received
0925 01:30:00.640665 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.640674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.640784 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.640876 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:00.643519 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.645018 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.647618 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.648539 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.648676 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.653977 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:00.653998 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:00.654027 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:00.654076 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:00.658118 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:00.659616 (+ 1498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:00.659626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:00.659631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:00.659794 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:00.660251 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:00.660256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:00.660260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:00.660724 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:00.661040 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.661433 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.661676 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 01:30:00.662410 3377 ts_tablet_manager.cc:1939] T 2a49847c982144209db4cc1f101bbd1d P 5b805c41709d45b1a9cd4d074a450adf: Deleting consensus metadata
I20260925 01:30:00.662963 2962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:00.663447 2774 catalog_manager.cc:4841] TS 5b805c41709d45b1a9cd4d074a450adf (127.27.160.3:33863): tablet 2a49847c982144209db4cc1f101bbd1d (table test-table [id=50b74a6535b646d2b0695e76982cf644]) successfully deleted
I20260925 01:30:00.782775 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 2769
I20260925 01:30:00.805761 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 2979
I20260925 01:30:00.821534 3350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.806627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.27.160.2:36355)
0925 01:30:00.806833 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.806839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.807009 (+ 170us) server_negotiation.cc:407] Connection header received
0925 01:30:00.807233 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.807238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.807332 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.807456 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:00.808848 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.809770 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.811175 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.811682 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.811838 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.814809 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:00.814821 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:00.814844 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:00.814886 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:00.818480 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:00.819215 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:00.819223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:00.819226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:00.819399 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:00.819814 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:00.819819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:00.819828 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:00.820217 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:00.820572 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.820887 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.821161 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:30:00.822248 3108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:00.944305 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 2979
I20260925 01:30:00.963922 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 3126
I20260925 01:30:00.977308 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:00.964679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.27.160.3:33863)
0925 01:30:00.964859 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:00.964866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:00.964956 (+ 90us) server_negotiation.cc:407] Connection header received
0925 01:30:00.965209 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:00.965219 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:00.965291 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:00.965393 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:00.966749 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.967487 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.968705 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:00.969098 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:00.969254 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:00.971760 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:00.971776 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:00.971793 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:00.971827 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:00.974564 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:00.975224 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:00.975232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:00.975234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:00.975354 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:00.975728 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:00.975732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:00.975735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:00.976097 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:00.976403 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:00.976698 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 01:30:00.976924 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:30:00.978091 3249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:01.014732 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:01.014317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42270)
0925 01:30:01.014488 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:01.014581 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 01:30:01.016003 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:01.015653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.0.0.1:42274)
0925 01:30:01.015797 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:01.015883 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:33983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 01:30:01.073974 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 3126
I20260925 01:30:01.091334 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 2459
I20260925 01:30:01.092391 2626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:01.210302 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 2459
W20260925 01:30:01.231562 2774 connection.cc:535] client connection to 127.27.160.61:37869 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 01:30:01.231801 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 2559
I20260925 01:30:01.232812 2818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:01.232858 3330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:01.232448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:37869 (local address 127.0.0.1:38216)
0925 01:30:01.232612 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:01.232709 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:30:01.354350 2774 consensus_peers.cc:487] T 00000000000000000000000000000000 P c8dcccfd51814f85b39ff6635a86ff4f -> Peer 4017e314eba74525b086cc1dcbd5e905 (127.27.160.61:37869): Couldn't send request to peer 4017e314eba74525b086cc1dcbd5e905. Status: Network error: Client connection negotiation failed: client connection to 127.27.160.61:37869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260925 01:30:01.354561 2774 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.27.160.61:37869, user_credentials={real_user=slave}} blocked reactor thread for 121494us
I20260925 01:30:01.354974 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 2559
2026-09-25T01:30:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9672 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:30:01 dist-test-slave-h6hf krb5kdc[3389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:30:01 dist-test-slave-h6hf krb5kdc[3389](info): set up 2 sockets
Sep 25 01:30:01 dist-test-slave-h6hf krb5kdc[3389](info): commencing operation
krb5kdc: starting...
W20260925 01:30:02.563637 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:30:02 dist-test-slave-h6hf krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:30:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:30:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.62.keytab.
Entry for principal oryx/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.62.keytab.
I20260925 01:30:02.797089 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:39315
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.62.keytab
--principal=oryx/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:39315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:03.126030 3402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:03.126406 3402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:03.126570 3402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:03.133984 3402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:30:03.134145 3402 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:30:03.134238 3402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:03.134354 3402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:03.134438 3402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:30:03.134537 3402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:30:03.144691 3402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:39315
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.62.keytab
--principal=oryx/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:39315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:03.146092 3402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:03.147456 3402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:03.160085 3408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:03.160037 3407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:03.160298 3410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:03.161252 3402 server_base.cc:1034] running on GCE node
I20260925 01:30:03.162258 3402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:03.164309 3402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:03.165570 3402 hybrid_clock.cc:648] HybridClock initialized: now 1790299803165506 us; error 36 us; skew 500 ppm
I20260925 01:30:03.165922 3402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:03 dist-test-slave-h6hf krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299803, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:03.169005 3402 init.cc:377] Logged in from keytab as oryx/127.27.160.62@KRBTEST.COM (short username oryx)
I20260925 01:30:03.170614 3402 webserver.cc:460] Webserver started at http://127.27.160.62:36455/ using document root <none> and password file <none>
I20260925 01:30:03.171371 3402 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:03.171514 3402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:03.171857 3402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:03.175127 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "524485c44abb4e0fa8527ae9cca62e89"
format_stamp: "Formatted at 2026-09-25 01:30:03 on dist-test-slave-h6hf"
server_key: "03dc0d9ea56c393a8271197d0856ecbb"
server_key_iv: "9b2bca42810c87d19a5f60f4c8ecbdb2"
server_key_version: "encryptionkey@0"
I20260925 01:30:03.175947 3402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "524485c44abb4e0fa8527ae9cca62e89"
format_stamp: "Formatted at 2026-09-25 01:30:03 on dist-test-slave-h6hf"
server_key: "03dc0d9ea56c393a8271197d0856ecbb"
server_key_iv: "9b2bca42810c87d19a5f60f4c8ecbdb2"
server_key_version: "encryptionkey@0"
I20260925 01:30:03.181260 3402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:30:03.184674 3417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:03.186036 3402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:03.186228 3402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "524485c44abb4e0fa8527ae9cca62e89"
format_stamp: "Formatted at 2026-09-25 01:30:03 on dist-test-slave-h6hf"
server_key: "03dc0d9ea56c393a8271197d0856ecbb"
server_key_iv: "9b2bca42810c87d19a5f60f4c8ecbdb2"
server_key_version: "encryptionkey@0"
I20260925 01:30:03.186453 3402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:03.207268 3402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:03.211341 3402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:03.211617 3402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:03.233260 3402 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:39315
I20260925 01:30:03.233291 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:39315 every 8 connection(s)
I20260925 01:30:03.234674 3402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:30:03.238399 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:03.238958 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 3402
I20260925 01:30:03.239199 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:30:03.240082 28288 external_mini_cluster.cc:1428] Setting key 29f627b48f461310a85b3357227cc691
I20260925 01:30:03.252969 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: Bootstrap starting.
Sep 25 01:30:03 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:30:03.256803 3479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:03.257993 3479 log.cc:824] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:03.260890 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: No bootstrap required, opened a new log
I20260925 01:30:03.263760 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:03.242294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.27.160.62:39315)
0925 01:30:03.243134 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:03.243147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:03.243189 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:30:03.244951 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:03.244976 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:03.245484 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:03.245843 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:03.247618 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.248931 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.250826 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.251525 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.251662 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:03.256328 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:03.256337 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:03.256353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:03.256392 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:03.259563 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:03.260295 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:03.260302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:03.260309 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:03.260484 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:03.260867 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:03.260871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:03.260873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:03.261529 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:03.261835 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:03.262272 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 01:30:03.262729 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":402,"threads_started":1}
I20260925 01:30:03.269229 3479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524485c44abb4e0fa8527ae9cca62e89" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39315 } }
I20260925 01:30:03.269616 3479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:03.269726 3479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524485c44abb4e0fa8527ae9cca62e89, State: Initialized, Role: FOLLOWER
I20260925 01:30:03.270319 3479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524485c44abb4e0fa8527ae9cca62e89" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39315 } }
I20260925 01:30:03.270566 3479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:30:03.270707 3479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:30:03.270879 3479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:03.273098 3479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524485c44abb4e0fa8527ae9cca62e89" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39315 } }
I20260925 01:30:03.273713 3479 leader_election.cc:302] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 524485c44abb4e0fa8527ae9cca62e89; no voters:
I20260925 01:30:03.274322 3479 leader_election.cc:288] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:30:03.274420 3484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:03.275804 3484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [term 1 LEADER]: Becoming Leader. State: Replica: 524485c44abb4e0fa8527ae9cca62e89, State: Running, Role: LEADER
I20260925 01:30:03.276361 3484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524485c44abb4e0fa8527ae9cca62e89" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39315 } }
I20260925 01:30:03.276624 3479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:30:03.279433 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 524485c44abb4e0fa8527ae9cca62e89. Latest consensus state: current_term: 1 leader_uuid: "524485c44abb4e0fa8527ae9cca62e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524485c44abb4e0fa8527ae9cca62e89" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39315 } } }
I20260925 01:30:03.279575 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "524485c44abb4e0fa8527ae9cca62e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524485c44abb4e0fa8527ae9cca62e89" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39315 } } }
I20260925 01:30:03.279861 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:03.279889 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89 [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:03.282025 3493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:30:03.286386 3493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:30:03.295293 3493 catalog_manager.cc:1347] Generated new cluster ID: eb29e1494e97483d98d34c18a6b252e1
I20260925 01:30:03.295425 3493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:30:03.305682 3493 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:30:03.307621 3493 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:30:03.318743 3493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: Generated new TSK 0
I20260925 01:30:03.319839 3493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab.
Entry for principal oryx/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab.
I20260925 01:30:03.384622 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39315
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab
--principal=oryx/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:03.733733 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:03.734033 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:03.734196 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:03.741623 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:03.741765 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:03.741922 3505 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:30:03.742049 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:30:03.752000 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab
--principal=oryx/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:39315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:03.753353 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:03.754856 3505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:03.767796 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:03.767892 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:03.767864 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:03.768740 3505 server_base.cc:1034] running on GCE node
I20260925 01:30:03.769454 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:03.770407 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:03.771703 3505 hybrid_clock.cc:648] HybridClock initialized: now 1790299803771628 us; error 43 us; skew 500 ppm
I20260925 01:30:03.772048 3505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:03 dist-test-slave-h6hf krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299803, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:03.775256 3505 init.cc:377] Logged in from keytab as oryx/127.27.160.1@KRBTEST.COM (short username oryx)
I20260925 01:30:03.776774 3505 webserver.cc:460] Webserver started at http://127.27.160.1:42201/ using document root <none> and password file <none>
I20260925 01:30:03.777566 3505 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:03.777694 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:03.778015 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:03.781019 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "b48fc46a2cc54054ab66353356ddb557"
format_stamp: "Formatted at 2026-09-25 01:30:03 on dist-test-slave-h6hf"
server_key: "0c27af5cf1af3159992d9b96947e9cd1"
server_key_iv: "ff74951bb7b8112c521337cb45ff1a71"
server_key_version: "encryptionkey@0"
I20260925 01:30:03.781888 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "b48fc46a2cc54054ab66353356ddb557"
format_stamp: "Formatted at 2026-09-25 01:30:03 on dist-test-slave-h6hf"
server_key: "0c27af5cf1af3159992d9b96947e9cd1"
server_key_iv: "ff74951bb7b8112c521337cb45ff1a71"
server_key_version: "encryptionkey@0"
I20260925 01:30:03.787091 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:30:03.790593 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:03.792026 3505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:30:03.792198 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "b48fc46a2cc54054ab66353356ddb557"
format_stamp: "Formatted at 2026-09-25 01:30:03 on dist-test-slave-h6hf"
server_key: "0c27af5cf1af3159992d9b96947e9cd1"
server_key_iv: "ff74951bb7b8112c521337cb45ff1a71"
server_key_version: "encryptionkey@0"
I20260925 01:30:03.792415 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:03.812953 3505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:03.817483 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:03.817790 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:03.820538 3505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:03.820675 3505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:03.820839 3505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:03.820916 3505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:30:03 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299803, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:30:03.844384 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:03.822687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:58995 (local address 127.27.160.62:39315)
0925 01:30:03.824156 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:03.824162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:03.824186 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:30:03.825439 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:03.825444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:03.825560 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:03.825675 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:03.827991 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.829343 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.831847 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.832468 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.832615 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:03.838271 (+ 5656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:03.838278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:03.838281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:03.838318 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:03.840957 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:03.841990 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:03.841994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:03.841996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:03.842121 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:03.842577 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:03.842581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:03.842585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:03.842856 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:03.843018 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:03.843779 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 01:30:03.844026 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":1249,"threads_started":1}
I20260925 01:30:03.846033 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:03.822192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39315 (local address 127.27.160.1:58995)
0925 01:30:03.822996 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:03.823051 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:03.825166 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:03.825967 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:03.825985 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:03.826870 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:03.827755 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:03.827792 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.829527 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.829534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:03.831673 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:03.831688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.832740 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.832749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:03.833039 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:03.834014 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:03.834078 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:03.837973 (+ 3895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:03.841139 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:03.841151 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:03.841171 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:03.841795 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:03.842237 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:03.842243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:03.842247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:03.842448 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:03.842986 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:03.842995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:03.843393 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 01:30:03.843677 (+ 284us) client_negotiation.cc:240] Negotiation successful
0925 01:30:03.844248 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":486,"threads_started":1}
I20260925 01:30:03.867476 3505 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:41635
I20260925 01:30:03.867496 3640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:41635 every 8 connection(s)
I20260925 01:30:03.869201 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:30:03.870631 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:03.853845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39315 (local address 127.27.160.1:37953)
0925 01:30:03.854513 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:03.854537 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:03.854746 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:03.855192 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:03.855199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:03.856089 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:03.856620 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:03.856642 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.858312 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.858317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:03.860147 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:03.860166 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.861557 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.861570 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:03.861717 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:03.862574 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:03.862598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:03.863367 (+ 769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:03.867646 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:03.867652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:03.867656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:03.868122 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:03.868539 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:03.868543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:03.868545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:03.868676 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:03.869351 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:03.869356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:03.869572 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 01:30:03.869690 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 01:30:03.870137 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":352,"threads_started":1}
I20260925 01:30:03.870641 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:03.853833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:37953 (local address 127.27.160.62:39315)
0925 01:30:03.854082 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:03.854090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:03.854651 (+ 561us) server_negotiation.cc:407] Connection header received
0925 01:30:03.854912 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:03.854920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:03.855026 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:03.855164 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:03.856857 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.858146 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.860356 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:03.861204 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:03.861410 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:03.863613 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:03.863619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:03.863625 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:03.863672 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:03.867492 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:03.868250 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:03.868259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:03.868265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:03.868425 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:03.868787 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:03.868792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:03.868795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:03.869218 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:03.869371 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:03.869866 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 01:30:03.870169 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:30:03.871711 3641 heartbeater.cc:347] Connected to a master server at 127.27.160.62:39315
I20260925 01:30:03.871941 3641 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:03.872807 3641 heartbeater.cc:511] Master 127.27.160.62:39315 requested a full tablet report, sending...
I20260925 01:30:03.875238 3434 ts_manager.cc:194] Registered new tserver with Master: b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635)
I20260925 01:30:03.876745 3434 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:58995
I20260925 01:30:03.878078 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 3505
I20260925 01:30:03.878281 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:30:03.878899 28288 external_mini_cluster.cc:1428] Setting key 260d8576db851b73b307b1bcbe54b6fb
WARNING: no policy specified for oryx/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab.
Entry for principal oryx/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab.
I20260925 01:30:03.925781 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39315
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab
--principal=oryx/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:04.251956 3645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:04.252246 3645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:04.252343 3645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:04.259720 3645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:04.259841 3645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:04.259927 3645 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:30:04.260000 3645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:30:04.270053 3645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab
--principal=oryx/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:39315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:04.271394 3645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:04.272725 3645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:04.286345 3650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:04.286415 3653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:04.286450 3651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:04.287357 3645 server_base.cc:1034] running on GCE node
I20260925 01:30:04.288017 3645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:04.288889 3645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:04.290114 3645 hybrid_clock.cc:648] HybridClock initialized: now 1790299804290026 us; error 56 us; skew 500 ppm
I20260925 01:30:04.290449 3645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:04 dist-test-slave-h6hf krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299804, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:04.293628 3645 init.cc:377] Logged in from keytab as oryx/127.27.160.2@KRBTEST.COM (short username oryx)
I20260925 01:30:04.295099 3645 webserver.cc:460] Webserver started at http://127.27.160.2:37657/ using document root <none> and password file <none>
I20260925 01:30:04.295815 3645 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:04.295944 3645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:04.296331 3645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:04.299319 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "30fbedf691a54c958677b7f2015a6076"
format_stamp: "Formatted at 2026-09-25 01:30:04 on dist-test-slave-h6hf"
server_key: "d4e7562f21fc93756c19b7d7f6b247f5"
server_key_iv: "8bca3773f6c83713a47a8545ee662a49"
server_key_version: "encryptionkey@0"
I20260925 01:30:04.300134 3645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "30fbedf691a54c958677b7f2015a6076"
format_stamp: "Formatted at 2026-09-25 01:30:04 on dist-test-slave-h6hf"
server_key: "d4e7562f21fc93756c19b7d7f6b247f5"
server_key_iv: "8bca3773f6c83713a47a8545ee662a49"
server_key_version: "encryptionkey@0"
I20260925 01:30:04.305394 3645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:30:04.308820 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:04.310245 3645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:04.310439 3645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "30fbedf691a54c958677b7f2015a6076"
format_stamp: "Formatted at 2026-09-25 01:30:04 on dist-test-slave-h6hf"
server_key: "d4e7562f21fc93756c19b7d7f6b247f5"
server_key_iv: "8bca3773f6c83713a47a8545ee662a49"
server_key_version: "encryptionkey@0"
I20260925 01:30:04.310662 3645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:04.329327 3645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:04.334831 3645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:04.335206 3645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:04.338021 3645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:04.338151 3645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:04.338320 3645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:04.338395 3645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:30:04 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299804, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:30:04.365936 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:04.338411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:60947 (local address 127.27.160.62:39315)
0925 01:30:04.338667 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:04.338677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:04.340163 (+ 1486us) server_negotiation.cc:407] Connection header received
0925 01:30:04.342303 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:04.342309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:04.342410 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:04.342550 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:04.344940 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.346056 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.348725 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.349256 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.349400 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:04.355040 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:04.355046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:04.355049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:04.355085 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:04.359012 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:04.360420 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:04.360428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:04.360435 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:04.360618 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:04.361351 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:04.361358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:04.361361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:04.361818 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:04.362000 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:04.365088 (+ 3088us) server_negotiation.cc:299] Negotiation successful
0925 01:30:04.365423 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 01:30:04.365943 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:04.339085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39315 (local address 127.27.160.2:60947)
0925 01:30:04.339964 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:04.340023 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:04.341961 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:04.342845 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:04.342858 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:04.343757 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:04.344669 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:04.344701 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.346229 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.346233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:04.348569 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:04.348581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.349524 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.349533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:04.349839 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:04.350761 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:04.350799 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:04.354794 (+ 3995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:04.359234 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:04.359249 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:04.359279 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:04.360204 (+ 925us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:04.360763 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:04.360768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:04.360772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:04.361166 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:04.362106 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:04.362121 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:04.362752 (+ 631us) client_negotiation.cc:769] Sending connection context
0925 01:30:04.363182 (+ 430us) client_negotiation.cc:240] Negotiation successful
0925 01:30:04.364010 (+ 828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":391,"threads_started":1}
I20260925 01:30:04.385689 3645 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:38753
I20260925 01:30:04.385746 3779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:38753 every 8 connection(s)
I20260925 01:30:04.386224 3530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:04.371486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:41869 (local address 127.27.160.62:39315)
0925 01:30:04.371729 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:04.371738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:04.372384 (+ 646us) server_negotiation.cc:407] Connection header received
0925 01:30:04.372603 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:04.372607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:04.372667 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:04.372740 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:04.374332 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.375048 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.376501 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.376957 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.377095 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:04.379728 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:04.379737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:04.379744 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:04.379789 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:04.383060 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:04.383913 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:04.383919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:04.383923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:04.384070 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:04.384425 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:04.384431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:04.384434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:04.384868 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:04.385046 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:04.385654 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 01:30:04.385894 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:30:04.386358 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:04.371486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39315 (local address 127.27.160.2:41869)
0925 01:30:04.372269 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:04.372294 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:04.372468 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:04.372817 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:04.372825 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:04.373637 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:04.374188 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:04.374201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.375239 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.375244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:04.376360 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:04.376372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.377193 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.377200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:04.377371 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:04.378225 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:04.378255 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:04.379470 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:04.383254 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:04.383260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:04.383265 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:04.383783 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:04.384179 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:04.384184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:04.384188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:04.384310 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:04.385014 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:04.385021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:04.385315 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 01:30:04.385543 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 01:30:04.385905 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":392,"threads_started":1}
I20260925 01:30:04.387315 3645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:30:04.389606 3780 heartbeater.cc:347] Connected to a master server at 127.27.160.62:39315
I20260925 01:30:04.389837 3780 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:04.390609 3780 heartbeater.cc:511] Master 127.27.160.62:39315 requested a full tablet report, sending...
I20260925 01:30:04.391919 3434 ts_manager.cc:194] Registered new tserver with Master: 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753)
I20260925 01:30:04.392722 3434 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:60947
I20260925 01:30:04.397157 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 3645
I20260925 01:30:04.397344 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:30:04.397804 28288 external_mini_cluster.cc:1428] Setting key fecd7c050bd6b95f46339dfddc986ddf
WARNING: no policy specified for oryx/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab.
Entry for principal oryx/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab.
I20260925 01:30:04.445307 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:39315
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab
--principal=oryx/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:39157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:04.790449 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:04.790778 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:04.790952 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:04.799407 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:04.799522 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:04.799666 3784 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:30:04.799762 3784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:30:04.810667 3784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab
--principal=oryx/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.160.62:39315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:04.812068 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:04.813532 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:04.828193 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:04.828153 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:04.828255 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:04.834378 3784 server_base.cc:1034] running on GCE node
I20260925 01:30:04.835160 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:04.836289 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:04.837546 3784 hybrid_clock.cc:648] HybridClock initialized: now 1790299804837476 us; error 42 us; skew 500 ppm
I20260925 01:30:04.837853 3784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:04 dist-test-slave-h6hf krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299804, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:04.841377 3784 init.cc:377] Logged in from keytab as oryx/127.27.160.3@KRBTEST.COM (short username oryx)
I20260925 01:30:04.842983 3784 webserver.cc:460] Webserver started at http://127.27.160.3:45863/ using document root <none> and password file <none>
I20260925 01:30:04.843750 3784 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:04.843865 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:04.844343 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:04.847795 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805"
format_stamp: "Formatted at 2026-09-25 01:30:04 on dist-test-slave-h6hf"
server_key: "eb651ae12e3be453fb2cef3a3fa17d8c"
server_key_iv: "841242e32af872020a3d6f84110a29e5"
server_key_version: "encryptionkey@0"
I20260925 01:30:04.848809 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805"
format_stamp: "Formatted at 2026-09-25 01:30:04 on dist-test-slave-h6hf"
server_key: "eb651ae12e3be453fb2cef3a3fa17d8c"
server_key_iv: "841242e32af872020a3d6f84110a29e5"
server_key_version: "encryptionkey@0"
I20260925 01:30:04.854669 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:30:04.858588 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:04.860095 3784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:30:04.860292 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805"
format_stamp: "Formatted at 2026-09-25 01:30:04 on dist-test-slave-h6hf"
server_key: "eb651ae12e3be453fb2cef3a3fa17d8c"
server_key_iv: "841242e32af872020a3d6f84110a29e5"
server_key_version: "encryptionkey@0"
I20260925 01:30:04.860563 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:04.879843 3641 heartbeater.cc:503] Master 127.27.160.62:39315 was elected leader, sending a full tablet report...
I20260925 01:30:04.892470 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:04.896669 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:04.896924 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:04.899636 3784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:04.899744 3784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:04.899907 3784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:04.899974 3784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:30:04 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299804, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:30:04.927089 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:04.900603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:42109 (local address 127.27.160.62:39315)
0925 01:30:04.901223 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:04.901228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:04.902435 (+ 1207us) server_negotiation.cc:407] Connection header received
0925 01:30:04.904688 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:04.904694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:04.904803 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:04.904938 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:04.907500 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.908744 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.912278 (+ 3534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.913154 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.913300 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:04.918549 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:04.918559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:04.918565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:04.918626 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:04.922083 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:04.923476 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:04.923484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:04.923489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:04.923664 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:04.924351 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:04.924358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:04.924364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:04.924794 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:04.924954 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:04.926299 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0925 01:30:04.926572 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":327,"threads_started":1}
I20260925 01:30:04.929416 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:04.901360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39315 (local address 127.27.160.3:42109)
0925 01:30:04.902251 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:04.902297 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:04.904368 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:04.905262 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:04.905270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:04.906240 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:04.907204 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:04.907240 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.908940 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.908945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:04.912086 (+ 3141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:04.912109 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.913488 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.913497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:04.913918 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:04.914761 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:04.914783 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:04.918222 (+ 3439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:04.922263 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:04.922278 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:04.922305 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:04.923309 (+ 1004us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:04.923810 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:04.923818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:04.923826 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:04.924210 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:04.924969 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:04.924983 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:04.925699 (+ 716us) client_negotiation.cc:769] Sending connection context
0925 01:30:04.926144 (+ 445us) client_negotiation.cc:240] Negotiation successful
0925 01:30:04.926941 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":382,"threads_started":1}
I20260925 01:30:04.944188 3784 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:44773
I20260925 01:30:04.944232 3919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:44773 every 8 connection(s)
I20260925 01:30:04.945803 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:30:04.948006 3920 heartbeater.cc:347] Connected to a master server at 127.27.160.62:39315
I20260925 01:30:04.948287 3920 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:04.948441 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 3784
I20260925 01:30:04.948637 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:30:04.949110 28288 external_mini_cluster.cc:1428] Setting key c14f30cb0411ce79d106c510158b57a6
I20260925 01:30:04.949152 3920 heartbeater.cc:511] Master 127.27.160.62:39315 requested a full tablet report, sending...
I20260925 01:30:04.950613 3434 ts_manager.cc:194] Registered new tserver with Master: fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:04.951663 3434 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:42109
I20260925 01:30:04.953015 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:04.937881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39315 (local address 127.27.160.3:39591)
0925 01:30:04.938542 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:04.938567 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:04.938764 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:04.939215 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:04.939219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:04.939653 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:04.939941 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:04.939950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.941574 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.941582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:04.943128 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:04.943149 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.944308 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.944316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:04.944511 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:04.945218 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:04.945235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:04.946030 (+ 795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:04.950282 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:04.950290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:04.950296 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:04.950714 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:04.951187 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:04.951192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:04.951196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:04.951325 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:04.951983 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:04.951988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:04.952166 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 01:30:04.952272 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 01:30:04.952564 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":364,"threads_started":1}
I20260925 01:30:04.953239 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:30:04.953348 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:04.937830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:39591 (local address 127.27.160.62:39315)
0925 01:30:04.938176 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:04.938186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:04.938678 (+ 492us) server_negotiation.cc:407] Connection header received
0925 01:30:04.938946 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:04.938955 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:04.939058 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:04.939181 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:04.940087 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.941408 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.943317 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.944077 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.944228 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:04.946237 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:04.946247 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:04.946253 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:04.946304 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:04.950160 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:04.950829 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:04.950835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:04.950840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:04.951047 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:04.951474 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:04.951482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:04.951486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:04.951842 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:04.952583 (+ 741us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:04.952718 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 01:30:04.952965 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 01:30:04 dist-test-slave-h6hf krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299804, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:30:04 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:04 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:05.395099 3780 heartbeater.cc:503] Master 127.27.160.62:39315 was elected leader, sending a full tablet report...
I20260925 01:30:05.954958 3920 heartbeater.cc:503] Master 127.27.160.62:39315 was elected leader, sending a full tablet report...
W20260925 01:30:06.163293 3928 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.164s user 0.006s sys 0.009s
W20260925 01:30:06.164628 3808 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:04.993484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.27.160.62:39315)
0925 01:30:04.993708 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:04.993713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:04.993977 (+ 264us) server_negotiation.cc:407] Connection header received
0925 01:30:04.994132 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:04.994136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:04.994204 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:04.994298 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:04.995642 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.996467 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.997753 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:04.998142 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:04.998235 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.164246 (+1166011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37030: BlockingRecv error: recv got EOF from 127.0.0.1:37030 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 01:30:06 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299804, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:30:06.184226 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.169028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.27.160.62:39315)
0925 01:30:06.169259 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.169264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.169642 (+ 378us) server_negotiation.cc:407] Connection header received
0925 01:30:06.169790 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.169794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.169862 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.169955 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:06.171135 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.171833 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.173689 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.174316 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.174475 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.178676 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:06.178682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:06.178686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:06.178724 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:06.181766 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:06.182459 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:06.182463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:06.182466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:06.182620 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:06.183011 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:06.183015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:06.183018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:06.183324 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:06.183554 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.183786 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.183984 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:30:06.188602 3433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:30:06.195693 3433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:30:06.221644 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.208851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:44773 (local address 127.0.0.1:38480)
0925 01:30:06.209633 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.209668 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.209863 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.210957 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.210970 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.211010 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:06.211453 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.211470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.213984 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.213993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.217287 (+ 3294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.217311 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.220248 (+ 2937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.220258 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.220559 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.220635 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.220817 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.221102 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":398,"threads_started":1}
I20260925 01:30:06.222924 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.209014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.27.160.3:44773)
0925 01:30:06.209615 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.209626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.209927 (+ 301us) server_negotiation.cc:407] Connection header received
0925 01:30:06.210144 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.210156 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.210412 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.210639 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:06.211681 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.213743 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.217545 (+ 3802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.219591 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.219763 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.220067 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.222240 (+ 2173us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.222504 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":293,"threads_started":1}
I20260925 01:30:06.228483 3850 tablet_service.cc:1467] Processing CreateTablet for tablet 93830b23dc1a4cc3a1ac7e1927e2338e (DEFAULT_TABLE table=test-table [id=b5e74c695e1443f9986d54ccc572339e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:06.230608 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93830b23dc1a4cc3a1ac7e1927e2338e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.232080 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.209990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:41635 (local address 127.0.0.1:43780)
0925 01:30:06.221950 (+ 11960us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.221969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.222079 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.224143 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.224152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.224180 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:06.224362 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.224370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.226122 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.226127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.228697 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.228721 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.231034 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.231045 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.231289 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.231360 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.231517 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.231694 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11821,"thread_start_us":20227,"threads_started":1}
I20260925 01:30:06.232179 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.210273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43780 (local address 127.27.160.1:41635)
0925 01:30:06.223568 (+ 13295us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.223575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.223599 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:30:06.223678 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.223684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.223883 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.224515 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:06.224678 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.225925 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.228991 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.230780 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.230971 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.231208 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.231684 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.231852 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13131,"thread_start_us":12999,"threads_started":1}
I20260925 01:30:06.236276 3571 tablet_service.cc:1467] Processing CreateTablet for tablet 93830b23dc1a4cc3a1ac7e1927e2338e (DEFAULT_TABLE table=test-table [id=b5e74c695e1443f9986d54ccc572339e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:06.237821 3571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93830b23dc1a4cc3a1ac7e1927e2338e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.240954 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.211481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.27.160.2:38753)
0925 01:30:06.228947 (+ 17466us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.228959 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.228998 (+ 39us) server_negotiation.cc:407] Connection header received
0925 01:30:06.229345 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.229356 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.229640 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.229872 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:06.231531 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.234062 (+ 2531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.236526 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.239457 (+ 2931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.239608 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.239831 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.240442 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.240633 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17273,"thread_start_us":17642,"threads_started":1}
I20260925 01:30:06.241925 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.210720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:38753 (local address 127.0.0.1:54366)
0925 01:30:06.223879 (+ 13159us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.223904 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.224048 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.230494 (+ 6446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.230503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.230533 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:06.231303 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.231318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.234229 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.234235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.236325 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.236342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.240034 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.240040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.240208 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.240256 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.241409 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.241588 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12991,"thread_start_us":13538,"threads_started":1}
I20260925 01:30:06.246140 3710 tablet_service.cc:1467] Processing CreateTablet for tablet 93830b23dc1a4cc3a1ac7e1927e2338e (DEFAULT_TABLE table=test-table [id=b5e74c695e1443f9986d54ccc572339e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:06.249416 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93830b23dc1a4cc3a1ac7e1927e2338e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.254581 3945 tablet_bootstrap.cc:492] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Bootstrap starting.
I20260925 01:30:06.260373 3945 tablet_bootstrap.cc:654] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.261430 3946 tablet_bootstrap.cc:492] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Bootstrap starting.
I20260925 01:30:06.262123 3945 log.cc:824] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:06.265631 3946 tablet_bootstrap.cc:654] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.266153 3945 tablet_bootstrap.cc:492] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: No bootstrap required, opened a new log
I20260925 01:30:06.266477 3945 ts_tablet_manager.cc:1397] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260925 01:30:06.267243 3946 log.cc:824] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:06.269919 3949 tablet_bootstrap.cc:492] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Bootstrap starting.
I20260925 01:30:06.270925 3946 tablet_bootstrap.cc:492] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: No bootstrap required, opened a new log
I20260925 01:30:06.271237 3946 ts_tablet_manager.cc:1397] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 01:30:06.273845 3949 tablet_bootstrap.cc:654] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.274948 3949 log.cc:824] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:06.278047 3949 tablet_bootstrap.cc:492] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: No bootstrap required, opened a new log
I20260925 01:30:06.278326 3949 ts_tablet_manager.cc:1397] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 01:30:06.279387 3945 raft_consensus.cc:348] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.279937 3945 raft_consensus.cc:374] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.280058 3945 raft_consensus.cc:729] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe60bb1ba7ef4cbeb3d885102b4b9805, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.280809 3945 consensus_queue.cc:260] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.283690 3946 raft_consensus.cc:348] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.284219 3946 raft_consensus.cc:374] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.284346 3946 raft_consensus.cc:729] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48fc46a2cc54054ab66353356ddb557, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.285089 3945 ts_tablet_manager.cc:1428] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent starting tablet: real 0.018s user 0.012s sys 0.006s
I20260925 01:30:06.285079 3946 consensus_queue.cc:260] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.287117 3946 ts_tablet_manager.cc:1428] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260925 01:30:06.288538 3949 raft_consensus.cc:348] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.289036 3949 raft_consensus.cc:374] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.289201 3949 raft_consensus.cc:729] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbedf691a54c958677b7f2015a6076, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.289786 3949 consensus_queue.cc:260] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.294456 3949 ts_tablet_manager.cc:1428] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260925 01:30:06.373903 3642 tablet.cc:2369] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:30:06.390762 3781 tablet.cc:2369] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:30:06.436936 3952 raft_consensus.cc:482] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:30:06.437353 3952 raft_consensus.cc:504] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.439124 3952 leader_election.cc:288] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773), 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753)
I20260925 01:30:06.446180 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.439639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:47497 (local address 127.27.160.3:44773)
0925 01:30:06.439908 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.439920 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.440848 (+ 928us) server_negotiation.cc:407] Connection header received
0925 01:30:06.441069 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.441075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.441197 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.441297 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:06.441868 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.442595 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.444366 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.445252 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.445419 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.445498 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.445762 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.445885 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:30:06.446218 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.439497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:44773 (local address 127.27.160.1:47497)
0925 01:30:06.440720 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.440742 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.440925 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.441311 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.441316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.441339 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:06.441725 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.441734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.442718 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.442726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.444215 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.444227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.445449 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.445455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.445613 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.445637 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.445739 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.445871 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":750,"threads_started":1}
I20260925 01:30:06.447108 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.439638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:52679 (local address 127.27.160.2:38753)
0925 01:30:06.439887 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.439896 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.440880 (+ 984us) server_negotiation.cc:407] Connection header received
0925 01:30:06.441198 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.441203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.441273 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.441390 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:06.442129 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.442911 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.444937 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.445914 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.446117 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.446213 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.446666 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.446811 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:30:06.447292 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.439563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:38753 (local address 127.27.160.1:52679)
0925 01:30:06.440728 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.440746 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.440993 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.441414 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.441419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.441465 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:06.441927 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.441938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.443077 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.443085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.444737 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.444752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.446172 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.446180 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.446458 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.446506 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.446637 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.446780 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":1,"spinlock_wait_cycles":27520,"thread_start_us":547,"threads_started":1}
I20260925 01:30:06.447636 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93830b23dc1a4cc3a1ac7e1927e2338e" candidate_uuid: "b48fc46a2cc54054ab66353356ddb557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" is_pre_election: true
I20260925 01:30:06.448194 3870 raft_consensus.cc:2454] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b48fc46a2cc54054ab66353356ddb557 in term 0.
I20260925 01:30:06.448539 3730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93830b23dc1a4cc3a1ac7e1927e2338e" candidate_uuid: "b48fc46a2cc54054ab66353356ddb557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fbedf691a54c958677b7f2015a6076" is_pre_election: true
W20260925 01:30:06.448935 3921 tablet.cc:2369] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:30:06.449218 3730 raft_consensus.cc:2454] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b48fc46a2cc54054ab66353356ddb557 in term 0.
I20260925 01:30:06.449218 3522 leader_election.cc:302] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b48fc46a2cc54054ab66353356ddb557, fe60bb1ba7ef4cbeb3d885102b4b9805; no voters:
I20260925 01:30:06.449726 3952 raft_consensus.cc:2781] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:30:06.449922 3952 raft_consensus.cc:482] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:30:06.450048 3952 raft_consensus.cc:3037] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.452412 3952 raft_consensus.cc:504] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.453053 3952 leader_election.cc:288] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [CANDIDATE]: Term 1 election: Requested vote from peers fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773), 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753)
I20260925 01:30:06.453567 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93830b23dc1a4cc3a1ac7e1927e2338e" candidate_uuid: "b48fc46a2cc54054ab66353356ddb557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805"
I20260925 01:30:06.453589 3730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93830b23dc1a4cc3a1ac7e1927e2338e" candidate_uuid: "b48fc46a2cc54054ab66353356ddb557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fbedf691a54c958677b7f2015a6076"
I20260925 01:30:06.453817 3730 raft_consensus.cc:3037] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.453817 3870 raft_consensus.cc:3037] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.457437 3730 raft_consensus.cc:2454] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48fc46a2cc54054ab66353356ddb557 in term 1.
I20260925 01:30:06.457428 3870 raft_consensus.cc:2454] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48fc46a2cc54054ab66353356ddb557 in term 1.
I20260925 01:30:06.457929 3522 leader_election.cc:302] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b48fc46a2cc54054ab66353356ddb557, fe60bb1ba7ef4cbeb3d885102b4b9805; no voters:
I20260925 01:30:06.458304 3952 raft_consensus.cc:2781] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:06.458791 3952 raft_consensus.cc:686] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 1 LEADER]: Becoming Leader. State: Replica: b48fc46a2cc54054ab66353356ddb557, State: Running, Role: LEADER
I20260925 01:30:06.459275 3952 consensus_queue.cc:237] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } }
I20260925 01:30:06.464470 3434 catalog_manager.cc:5495] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 reported cstate change: term changed from 0 to 1, leader changed from <none> to b48fc46a2cc54054ab66353356ddb557 (127.27.160.1). New cstate: current_term: 1 leader_uuid: "b48fc46a2cc54054ab66353356ddb557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } health_report { overall_health: UNKNOWN } } }
I20260925 01:30:06.518740 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.512461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.27.160.1:41635)
0925 01:30:06.512832 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.512840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.513008 (+ 168us) server_negotiation.cc:407] Connection header received
0925 01:30:06.513340 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.513345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.513444 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.513641 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:06.514071 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.514858 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.516293 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.516725 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.516881 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.517466 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:06.518056 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:06.518236 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.518341 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.518437 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260925 01:30:06.527149 3730 raft_consensus.cc:1261] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Refusing update from remote peer b48fc46a2cc54054ab66353356ddb557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:06.527149 3870 raft_consensus.cc:1261] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Refusing update from remote peer b48fc46a2cc54054ab66353356ddb557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:06.528228 3956 consensus_queue.cc:1035] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:06.528587 3952 consensus_queue.cc:1035] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:06.541455 3960 mvcc.cc:205] Tried to move back new op lower bound from 7333068007523995648 to 7333068007260626944. Current Snapshot: MvccSnapshot[applied={T|T < 7333068007523995648}]
I20260925 01:30:06.560897 3432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:06.561311 3432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:06.563251 3961 mvcc.cc:205] Tried to move back new op lower bound from 7333068007523995648 to 7333068007260626944. Current Snapshot: MvccSnapshot[applied={T|T < 7333068007523995648}]
I20260925 01:30:06.566697 3432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: Sending DeleteTablet for 3 replicas of tablet 93830b23dc1a4cc3a1ac7e1927e2338e
I20260925 01:30:06.568027 3571 tablet_service.cc:1514] Processing DeleteTablet for tablet 93830b23dc1a4cc3a1ac7e1927e2338e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:06 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:43780
I20260925 01:30:06.568375 3710 tablet_service.cc:1514] Processing DeleteTablet for tablet 93830b23dc1a4cc3a1ac7e1927e2338e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:06 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:54366
I20260925 01:30:06.569766 3850 tablet_service.cc:1514] Processing DeleteTablet for tablet 93830b23dc1a4cc3a1ac7e1927e2338e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:06 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:38480
I20260925 01:30:06.570631 3709 tablet_service.cc:1514] Processing DeleteTablet for tablet 93830b23dc1a4cc3a1ac7e1927e2338e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:06 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:54366
I20260925 01:30:06.571542 3570 tablet_service.cc:1514] Processing DeleteTablet for tablet 93830b23dc1a4cc3a1ac7e1927e2338e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:06 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:43780
I20260925 01:30:06.574056 3432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:30:06.575091 3432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:30:06.578279 3971 tablet_replica.cc:330] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: stopping tablet replica
I20260925 01:30:06.578779 3971 raft_consensus.cc:2229] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:06.580500 3971 raft_consensus.cc:2258] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:06.583393 3971 ts_tablet_manager.cc:1905] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:06.586858 3709 tablet_service.cc:1467] Processing CreateTablet for tablet 06802f13b4b74bbea57ecc455435f935 (DEFAULT_TABLE table=test-table [id=8ed557a6e1b840929bff3b0187f50423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:06.587502 3709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06802f13b4b74bbea57ecc455435f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.587560 3570 tablet_service.cc:1467] Processing CreateTablet for tablet 06802f13b4b74bbea57ecc455435f935 (DEFAULT_TABLE table=test-table [id=8ed557a6e1b840929bff3b0187f50423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:06.588366 3570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06802f13b4b74bbea57ecc455435f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.593931 3850 tablet_service.cc:1467] Processing CreateTablet for tablet 06802f13b4b74bbea57ecc455435f935 (DEFAULT_TABLE table=test-table [id=8ed557a6e1b840929bff3b0187f50423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:06.594532 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06802f13b4b74bbea57ecc455435f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.594643 3949 tablet_bootstrap.cc:492] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: Bootstrap starting.
I20260925 01:30:06.596580 3949 tablet_bootstrap.cc:654] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.600001 3946 tablet_bootstrap.cc:492] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: Bootstrap starting.
I20260925 01:30:06.601727 3946 tablet_bootstrap.cc:654] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.603241 3945 tablet_bootstrap.cc:492] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: Bootstrap starting.
I20260925 01:30:06.605427 3945 tablet_bootstrap.cc:654] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.607375 3970 tablet_replica.cc:330] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: stopping tablet replica
I20260925 01:30:06.607883 3970 raft_consensus.cc:2229] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:06.608415 3970 raft_consensus.cc:2258] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:06.609953 3949 tablet_bootstrap.cc:492] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: No bootstrap required, opened a new log
I20260925 01:30:06.610105 3949 ts_tablet_manager.cc:1397] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20260925 01:30:06.610432 3945 tablet_bootstrap.cc:492] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: No bootstrap required, opened a new log
I20260925 01:30:06.610601 3945 ts_tablet_manager.cc:1397] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260925 01:30:06.610941 3949 raft_consensus.cc:348] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.611178 3949 raft_consensus.cc:374] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.611279 3949 raft_consensus.cc:729] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbedf691a54c958677b7f2015a6076, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.611588 3949 consensus_queue.cc:260] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.611639 3945 raft_consensus.cc:348] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.611914 3945 raft_consensus.cc:374] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.612033 3945 raft_consensus.cc:729] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe60bb1ba7ef4cbeb3d885102b4b9805, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.613865 3949 ts_tablet_manager.cc:1428] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260925 01:30:06.616751 3970 ts_tablet_manager.cc:1905] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:06.617403 3969 tablet_replica.cc:330] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: stopping tablet replica
I20260925 01:30:06.617991 3969 raft_consensus.cc:2229] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:30:06.613833 3945 consensus_queue.cc:260] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.619226 3945 ts_tablet_manager.cc:1428] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260925 01:30:06.619426 3969 raft_consensus.cc:2258] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:06.622115 3946 tablet_bootstrap.cc:492] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: No bootstrap required, opened a new log
I20260925 01:30:06.622287 3946 ts_tablet_manager.cc:1397] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.007s
I20260925 01:30:06.623005 3971 ts_tablet_manager.cc:1918] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:06.623209 3971 log.cc:1196] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/93830b23dc1a4cc3a1ac7e1927e2338e
I20260925 01:30:06.623242 3946 raft_consensus.cc:348] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.623502 3946 raft_consensus.cc:374] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.623622 3946 raft_consensus.cc:729] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48fc46a2cc54054ab66353356ddb557, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.623895 3971 ts_tablet_manager.cc:1939] T 93830b23dc1a4cc3a1ac7e1927e2338e P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting consensus metadata
I20260925 01:30:06.623914 3946 consensus_queue.cc:260] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.624583 3946 ts_tablet_manager.cc:1428] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 01:30:06.625418 3969 ts_tablet_manager.cc:1905] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:06.626047 3419 catalog_manager.cc:4841] TS fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773): tablet 93830b23dc1a4cc3a1ac7e1927e2338e (table test-table [id=b5e74c695e1443f9986d54ccc572339e]) successfully deleted
I20260925 01:30:06.630935 3970 ts_tablet_manager.cc:1918] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:06.631134 3970 log.cc:1196] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/93830b23dc1a4cc3a1ac7e1927e2338e
I20260925 01:30:06.631718 3970 ts_tablet_manager.cc:1939] T 93830b23dc1a4cc3a1ac7e1927e2338e P 30fbedf691a54c958677b7f2015a6076: Deleting consensus metadata
I20260925 01:30:06.632918 3969 ts_tablet_manager.cc:1918] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:06.633101 3969 log.cc:1196] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/93830b23dc1a4cc3a1ac7e1927e2338e
I20260925 01:30:06.633603 3420 catalog_manager.cc:4841] TS 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753): tablet 93830b23dc1a4cc3a1ac7e1927e2338e (table test-table [id=b5e74c695e1443f9986d54ccc572339e]) successfully deleted
I20260925 01:30:06.633795 3969 ts_tablet_manager.cc:1939] T 93830b23dc1a4cc3a1ac7e1927e2338e P b48fc46a2cc54054ab66353356ddb557: Deleting consensus metadata
W20260925 01:30:06.634287 3420 catalog_manager.cc:4808] TS 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753): delete failed for tablet 93830b23dc1a4cc3a1ac7e1927e2338e because the tablet was not found. No further retry: Not found: Tablet not found: 93830b23dc1a4cc3a1ac7e1927e2338e
I20260925 01:30:06.635464 3418 catalog_manager.cc:4841] TS b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635): tablet 93830b23dc1a4cc3a1ac7e1927e2338e (table test-table [id=b5e74c695e1443f9986d54ccc572339e]) successfully deleted
W20260925 01:30:06.635872 3418 catalog_manager.cc:4808] TS b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635): delete failed for tablet 93830b23dc1a4cc3a1ac7e1927e2338e because the tablet was not found. No further retry: Not found: Tablet not found: 93830b23dc1a4cc3a1ac7e1927e2338e
I20260925 01:30:06.741523 3953 raft_consensus.cc:482] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:30:06.741771 3953 raft_consensus.cc:504] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.743525 3953 leader_election.cc:288] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635), fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:06.751901 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.744144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:44719 (local address 127.27.160.1:41635)
0925 01:30:06.744385 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.744392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.744743 (+ 351us) server_negotiation.cc:407] Connection header received
0925 01:30:06.745033 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.745039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.745159 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.745248 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:06.745958 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.746681 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.749614 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.750922 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.751095 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.751175 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.751431 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.751563 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 01:30:06.751935 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.743773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:41635 (local address 127.27.160.2:44719)
0925 01:30:06.744627 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.744649 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.744870 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.745328 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.745332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.745352 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:06.745813 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.745822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.746819 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.746828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.749455 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.749464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.751127 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.751132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.751252 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.751278 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.751389 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.751536 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":89,"thread_start_us":783,"threads_started":1}
I20260925 01:30:06.752954 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06802f13b4b74bbea57ecc455435f935" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48fc46a2cc54054ab66353356ddb557" is_pre_election: true
I20260925 01:30:06.753463 3591 raft_consensus.cc:2454] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 0.
I20260925 01:30:06.753763 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.744056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:54963 (local address 127.27.160.3:44773)
0925 01:30:06.744224 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.744231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.744651 (+ 420us) server_negotiation.cc:407] Connection header received
0925 01:30:06.744922 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.744929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.745016 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.745115 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:06.745760 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.746867 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.751256 (+ 4389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.752502 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.752677 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.752749 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.753298 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.753448 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 01:30:06.754271 3661 leader_election.cc:302] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fbedf691a54c958677b7f2015a6076, b48fc46a2cc54054ab66353356ddb557; no voters:
I20260925 01:30:06.754630 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.743763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:44773 (local address 127.27.160.2:54963)
0925 01:30:06.744540 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.744570 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.744758 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.745162 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.745166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.745192 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:06.745555 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.745566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.746996 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.747011 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.749448 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.749464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.752895 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.752902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.753084 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.753140 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.754012 (+ 872us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.754159 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":68,"thread_start_us":639,"threads_started":1}
I20260925 01:30:06.754719 3953 raft_consensus.cc:2781] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:30:06.754853 3953 raft_consensus.cc:482] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:30:06.755096 3953 raft_consensus.cc:3037] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.755445 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06802f13b4b74bbea57ecc455435f935" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" is_pre_election: true
I20260925 01:30:06.755699 3870 raft_consensus.cc:2454] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 0.
I20260925 01:30:06.757812 3953 raft_consensus.cc:504] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.758494 3953 leader_election.cc:288] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 election: Requested vote from peers b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635), fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:06.758982 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06802f13b4b74bbea57ecc455435f935" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48fc46a2cc54054ab66353356ddb557"
I20260925 01:30:06.759035 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06802f13b4b74bbea57ecc455435f935" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805"
I20260925 01:30:06.759227 3591 raft_consensus.cc:3037] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.759219 3870 raft_consensus.cc:3037] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.762326 3870 raft_consensus.cc:2454] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 1.
I20260925 01:30:06.762746 3591 raft_consensus.cc:2454] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 1.
I20260925 01:30:06.762877 3662 leader_election.cc:302] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fbedf691a54c958677b7f2015a6076, fe60bb1ba7ef4cbeb3d885102b4b9805; no voters:
I20260925 01:30:06.763415 3953 raft_consensus.cc:2781] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:06.763968 3953 raft_consensus.cc:686] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 1 LEADER]: Becoming Leader. State: Replica: 30fbedf691a54c958677b7f2015a6076, State: Running, Role: LEADER
I20260925 01:30:06.764628 3953 consensus_queue.cc:237] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.768484 3433 catalog_manager.cc:5495] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30fbedf691a54c958677b7f2015a6076 (127.27.160.2). New cstate: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } health_report { overall_health: UNKNOWN } } }
Sep 25 01:30:06 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299803, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.62@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:30:06.826110 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.807654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:39315 (local address 127.0.0.1:37048)
0925 01:30:06.808374 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.808403 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.808578 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.808978 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.808985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.809774 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:06.810014 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.810024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.811146 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.811153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.812108 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.812120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.812863 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.812868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.813072 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.817723 (+ 4651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:06.817755 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:06.820233 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:06.823457 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:06.823473 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:06.823481 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:06.823899 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:06.824272 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:06.824278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:06.824282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:06.824413 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:06.824935 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:06.824943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:06.825189 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.825365 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.825686 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":385,"threads_started":1}
I20260925 01:30:06.826172 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.807693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.27.160.62:39315)
0925 01:30:06.808705 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.808712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.808735 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:30:06.808774 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.808777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.808832 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.808944 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:06.810189 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.811002 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.812252 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.812687 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.812816 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.820494 (+ 7678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:06.820506 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:06.820510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:06.820543 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:06.823305 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:06.824027 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:06.824033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:06.824037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:06.824162 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:06.824523 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:06.824528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:06.824531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:06.824802 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:06.824974 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.825453 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.825817 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":897,"thread_start_us":677,"threads_started":1}
I20260925 01:30:06.831557 3433 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:37048:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:30:06.832705 3433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:30:06.840314 3571 tablet_service.cc:1467] Processing CreateTablet for tablet 3d265105feb74ec7af3bdb3fd4b2f3b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2515523e8b0e47a592f95208bdf5af06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:30:06.840364 3710 tablet_service.cc:1467] Processing CreateTablet for tablet 3d265105feb74ec7af3bdb3fd4b2f3b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2515523e8b0e47a592f95208bdf5af06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:30:06.840520 3850 tablet_service.cc:1467] Processing CreateTablet for tablet 3d265105feb74ec7af3bdb3fd4b2f3b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2515523e8b0e47a592f95208bdf5af06]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:30:06.840993 3571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d265105feb74ec7af3bdb3fd4b2f3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.840991 3710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d265105feb74ec7af3bdb3fd4b2f3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.841012 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d265105feb74ec7af3bdb3fd4b2f3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:06.848716 3949 tablet_bootstrap.cc:492] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: Bootstrap starting.
I20260925 01:30:06.848802 3945 tablet_bootstrap.cc:492] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: Bootstrap starting.
I20260925 01:30:06.850594 3945 tablet_bootstrap.cc:654] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.850594 3949 tablet_bootstrap.cc:654] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.851605 3946 tablet_bootstrap.cc:492] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: Bootstrap starting.
I20260925 01:30:06.853592 3946 tablet_bootstrap.cc:654] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:06.861654 3949 tablet_bootstrap.cc:492] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: No bootstrap required, opened a new log
I20260925 01:30:06.861901 3949 ts_tablet_manager.cc:1397] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260925 01:30:06.862811 3949 raft_consensus.cc:348] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.863098 3949 raft_consensus.cc:374] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.863191 3949 raft_consensus.cc:729] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbedf691a54c958677b7f2015a6076, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.863511 3949 consensus_queue.cc:260] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.863699 3945 tablet_bootstrap.cc:492] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: No bootstrap required, opened a new log
I20260925 01:30:06.863863 3945 ts_tablet_manager.cc:1397] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260925 01:30:06.863973 3953 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:06.864277 3953 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:30:06.864724 3945 raft_consensus.cc:348] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.864974 3945 raft_consensus.cc:374] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.865159 3945 raft_consensus.cc:729] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe60bb1ba7ef4cbeb3d885102b4b9805, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.865459 3945 consensus_queue.cc:260] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.866127 3945 ts_tablet_manager.cc:1428] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 01:30:06.865990 3951 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:06.866393 3951 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:30:06.867056 3949 ts_tablet_manager.cc:1428] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260925 01:30:06.870790 3946 tablet_bootstrap.cc:492] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: No bootstrap required, opened a new log
I20260925 01:30:06.871083 3946 ts_tablet_manager.cc:1397] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.008s
I20260925 01:30:06.871932 3946 raft_consensus.cc:348] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.872249 3946 raft_consensus.cc:374] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:06.872344 3946 raft_consensus.cc:729] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48fc46a2cc54054ab66353356ddb557, State: Initialized, Role: FOLLOWER
I20260925 01:30:06.872674 3946 consensus_queue.cc:260] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.873200 3956 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:06.873566 3956 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:30:06.874068 3946 ts_tablet_manager.cc:1428] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 01:30:06.925597 3953 raft_consensus.cc:482] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:30:06.925869 3953 raft_consensus.cc:504] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.926633 3953 leader_election.cc:288] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635), fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:06.927309 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d265105feb74ec7af3bdb3fd4b2f3b7" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" is_pre_election: true
I20260925 01:30:06.927340 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d265105feb74ec7af3bdb3fd4b2f3b7" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48fc46a2cc54054ab66353356ddb557" is_pre_election: true
I20260925 01:30:06.927592 3870 raft_consensus.cc:2454] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 0.
I20260925 01:30:06.927627 3591 raft_consensus.cc:2454] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 0.
I20260925 01:30:06.928162 3662 leader_election.cc:302] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30fbedf691a54c958677b7f2015a6076, b48fc46a2cc54054ab66353356ddb557, fe60bb1ba7ef4cbeb3d885102b4b9805; no voters:
I20260925 01:30:06.928550 3953 raft_consensus.cc:2781] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:30:06.928673 3953 raft_consensus.cc:482] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:30:06.928788 3953 raft_consensus.cc:3037] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.931898 3953 raft_consensus.cc:504] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.932660 3953 leader_election.cc:288] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 election: Requested vote from peers b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635), fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:06.933271 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d265105feb74ec7af3bdb3fd4b2f3b7" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805"
I20260925 01:30:06.933538 3870 raft_consensus.cc:3037] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.936585 3870 raft_consensus.cc:2454] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 1.
I20260925 01:30:06.937083 3662 leader_election.cc:302] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fbedf691a54c958677b7f2015a6076, fe60bb1ba7ef4cbeb3d885102b4b9805; no voters:
I20260925 01:30:06.937059 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d265105feb74ec7af3bdb3fd4b2f3b7" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48fc46a2cc54054ab66353356ddb557"
I20260925 01:30:06.937435 3591 raft_consensus.cc:3037] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:06.937438 3953 raft_consensus.cc:2781] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:06.937609 3953 raft_consensus.cc:686] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [term 1 LEADER]: Becoming Leader. State: Replica: 30fbedf691a54c958677b7f2015a6076, State: Running, Role: LEADER
I20260925 01:30:06.937886 3953 consensus_queue.cc:237] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:06.939692 3977 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: TxnStatusTablet state changed. Reason: New leader 30fbedf691a54c958677b7f2015a6076. Latest consensus state: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:06.940104 3977 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:30:06.940954 3591 raft_consensus.cc:2454] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 1.
I20260925 01:30:06.942126 3990 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:30:06.942386 3990 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:30:06.942246 3433 catalog_manager.cc:5495] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30fbedf691a54c958677b7f2015a6076 (127.27.160.2). New cstate: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } health_report { overall_health: UNKNOWN } } }
I20260925 01:30:06.992347 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.970094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:38753 (local address 127.0.0.1:54376)
0925 01:30:06.970936 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:06.970958 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:06.971118 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:06.984625 (+ 13507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:06.984633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:06.984649 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:30:06.984879 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.984891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.986800 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.986807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.988736 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:06.988754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.990032 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.990039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:06.990258 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.990375 (+ 117us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:30:06.991717 (+ 1342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:30:06.991744 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 01:30:06.991832 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 01:30:06.992010 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639}
I20260925 01:30:06.992364 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:06.971425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.27.160.2:38753)
0925 01:30:06.983937 (+ 12512us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:06.983946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:06.983974 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:30:06.984094 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:06.984100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:06.984221 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:06.985038 (+ 817us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:06.985271 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.986584 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.988943 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:06.989834 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:06.990600 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:06.990837 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:06.991575 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:06.991745 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:06.991869 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 01:30:06.991961 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12349,"thread_start_us":13013,"threads_started":1}
I20260925 01:30:06.996708 3870 raft_consensus.cc:1261] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Refusing update from remote peer 30fbedf691a54c958677b7f2015a6076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:06.997187 3591 raft_consensus.cc:1261] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Refusing update from remote peer 30fbedf691a54c958677b7f2015a6076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:06.997733 3977 consensus_queue.cc:1035] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:06.998759 3977 consensus_queue.cc:1035] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:07.006239 3956 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: TxnStatusTablet state changed. Reason: New leader 30fbedf691a54c958677b7f2015a6076. Latest consensus state: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:07.006639 3956 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:30:07.008947 3977 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:07.009243 3977 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:30:07.013350 3956 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:07.013686 3956 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P b48fc46a2cc54054ab66353356ddb557: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:30:07.020391 3951 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: TxnStatusTablet state changed. Reason: New leader 30fbedf691a54c958677b7f2015a6076. Latest consensus state: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:07.020843 3951 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:30:07.023342 3953 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:07.023792 3953 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P 30fbedf691a54c958677b7f2015a6076: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:30:07.029372 3951 tablet_replica.cc:439] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } } }
I20260925 01:30:07.029732 3951 tablet_replica.cc:442] T 3d265105feb74ec7af3bdb3fd4b2f3b7 P fe60bb1ba7ef4cbeb3d885102b4b9805: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:30:07.044639 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.031253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.27.160.2:38753)
0925 01:30:07.033324 (+ 2071us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.033337 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.037862 (+ 4525us) server_negotiation.cc:407] Connection header received
0925 01:30:07.038073 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.038079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.038166 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.038265 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:07.038910 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.040099 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.042366 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.042988 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.043151 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.043713 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.043881 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.044002 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.044187 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.044277 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1438}
I20260925 01:30:07.065204 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.057386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:47653 (local address 127.27.160.2:38753)
0925 01:30:07.057583 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.057589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.058353 (+ 764us) server_negotiation.cc:407] Connection header received
0925 01:30:07.058527 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.058534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.058647 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.058771 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:07.059350 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.060716 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.062813 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.063439 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.063532 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.064160 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.064309 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.064414 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.064575 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.064666 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:30:07.065172 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.057491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:38753 (local address 127.27.160.2:47653)
0925 01:30:07.058229 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:07.058252 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:07.058391 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:07.058777 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:07.058782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:07.058794 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:30:07.059177 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:07.059187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.060871 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.060877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:07.062666 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:07.062681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.063707 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.063713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:07.063871 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.063940 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.064439 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.064450 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:30:07.064535 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 01:30:07.064679 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":404,"threads_started":1}
I20260925 01:30:07.078231 3870 raft_consensus.cc:1261] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Refusing update from remote peer 30fbedf691a54c958677b7f2015a6076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:07.078394 3591 raft_consensus.cc:1261] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Refusing update from remote peer 30fbedf691a54c958677b7f2015a6076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:07.078755 3977 consensus_queue.cc:1035] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:07.079125 3953 consensus_queue.cc:1035] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:07.099846 3959 mvcc.cc:205] Tried to move back new op lower bound from 7333068009787326464 to 7333068008511950848. Current Snapshot: MvccSnapshot[applied={T|T < 7333068009787326464}]
I20260925 01:30:07.146101 3432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:07.146353 3432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:07.149468 3432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: Sending DeleteTablet for 3 replicas of tablet 06802f13b4b74bbea57ecc455435f935
I20260925 01:30:07.150312 3571 tablet_service.cc:1514] Processing DeleteTablet for tablet 06802f13b4b74bbea57ecc455435f935 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:43780
I20260925 01:30:07.150328 3709 tablet_service.cc:1514] Processing DeleteTablet for tablet 06802f13b4b74bbea57ecc455435f935 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:54366
I20260925 01:30:07.151515 3850 tablet_service.cc:1514] Processing DeleteTablet for tablet 06802f13b4b74bbea57ecc455435f935 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:38480
I20260925 01:30:07.152340 4007 tablet_replica.cc:330] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: stopping tablet replica
I20260925 01:30:07.152571 4007 raft_consensus.cc:2229] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:07.152793 4007 raft_consensus.cc:2258] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.153725 4007 ts_tablet_manager.cc:1905] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.161654 4009 tablet_replica.cc:330] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: stopping tablet replica
I20260925 01:30:07.161804 4008 tablet_replica.cc:330] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: stopping tablet replica
I20260925 01:30:07.161876 4009 raft_consensus.cc:2229] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:07.162110 4009 raft_consensus.cc:2258] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.162156 4008 raft_consensus.cc:2229] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:30:07.163167 4009 ts_tablet_manager.cc:1905] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.163774 4008 raft_consensus.cc:2258] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.164922 4008 ts_tablet_manager.cc:1905] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.171216 4007 ts_tablet_manager.cc:1918] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:30:07.171361 4007 log.cc:1196] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/06802f13b4b74bbea57ecc455435f935
I20260925 01:30:07.171792 4007 ts_tablet_manager.cc:1939] T 06802f13b4b74bbea57ecc455435f935 P b48fc46a2cc54054ab66353356ddb557: Deleting consensus metadata
I20260925 01:30:07.172129 4008 ts_tablet_manager.cc:1918] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:30:07.172281 4008 log.cc:1196] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/06802f13b4b74bbea57ecc455435f935
I20260925 01:30:07.172796 4008 ts_tablet_manager.cc:1939] T 06802f13b4b74bbea57ecc455435f935 P 30fbedf691a54c958677b7f2015a6076: Deleting consensus metadata
I20260925 01:30:07.173079 3418 catalog_manager.cc:4841] TS b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635): tablet 06802f13b4b74bbea57ecc455435f935 (table test-table [id=8ed557a6e1b840929bff3b0187f50423]) successfully deleted
I20260925 01:30:07.174036 3420 catalog_manager.cc:4841] TS 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753): tablet 06802f13b4b74bbea57ecc455435f935 (table test-table [id=8ed557a6e1b840929bff3b0187f50423]) successfully deleted
I20260925 01:30:07.181599 4009 ts_tablet_manager.cc:1918] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:30:07.181842 4009 log.cc:1196] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/06802f13b4b74bbea57ecc455435f935
I20260925 01:30:07.182334 4009 ts_tablet_manager.cc:1939] T 06802f13b4b74bbea57ecc455435f935 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting consensus metadata
I20260925 01:30:07.183565 3419 catalog_manager.cc:4841] TS fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773): tablet 06802f13b4b74bbea57ecc455435f935 (table test-table [id=8ed557a6e1b840929bff3b0187f50423]) successfully deleted
Sep 25 01:30:07 dist-test-slave-h6hf krb5kdc[3389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:30:07 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:07 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:30:07.219334 3985 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:07.207882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.27.160.62:39315)
0925 01:30:07.208096 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.208106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.208496 (+ 390us) server_negotiation.cc:407] Connection header received
0925 01:30:07.208734 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.208739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.208805 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.208912 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:07.210260 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.210995 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.212278 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.212716 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.212854 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.219078 (+ 6224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37064: BlockingRecv error: recv got EOF from 127.0.0.1:37064 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 01:30:07 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM
I20260925 01:30:07.237727 3985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.222381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.27.160.62:39315)
0925 01:30:07.222600 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.222608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.222828 (+ 220us) server_negotiation.cc:407] Connection header received
0925 01:30:07.222982 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.222988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.223070 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.223181 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:07.224476 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.225655 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.227043 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.227672 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.227784 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.231886 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:07.231892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:07.231896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:07.231933 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:07.234871 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:07.235541 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:07.235547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:07.235551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:07.235718 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:07.236111 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:07.236117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:07.236120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:07.236417 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:07.236694 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.236983 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.237250 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:30:07.241384 3432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:30:07.242067 3432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:30:07.249570 3709 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec2167890614ab69e55cdd21909d642 (DEFAULT_TABLE table=test-table [id=6195e171c5204b93bf913a7c77e7f66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:07.249706 3571 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec2167890614ab69e55cdd21909d642 (DEFAULT_TABLE table=test-table [id=6195e171c5204b93bf913a7c77e7f66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:07.250034 3709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec2167890614ab69e55cdd21909d642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:07.250182 3571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec2167890614ab69e55cdd21909d642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:07.252918 3850 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec2167890614ab69e55cdd21909d642 (DEFAULT_TABLE table=test-table [id=6195e171c5204b93bf913a7c77e7f66b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:07.253365 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec2167890614ab69e55cdd21909d642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:07.255959 3946 tablet_bootstrap.cc:492] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: Bootstrap starting.
I20260925 01:30:07.257381 3946 tablet_bootstrap.cc:654] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:07.259774 3946 tablet_bootstrap.cc:492] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: No bootstrap required, opened a new log
I20260925 01:30:07.259935 3946 ts_tablet_manager.cc:1397] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 01:30:07.260587 3946 raft_consensus.cc:348] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.260809 3946 raft_consensus.cc:374] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:07.260895 3946 raft_consensus.cc:729] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48fc46a2cc54054ab66353356ddb557, State: Initialized, Role: FOLLOWER
I20260925 01:30:07.261188 3946 consensus_queue.cc:260] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.261785 3946 ts_tablet_manager.cc:1428] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 01:30:07.264810 3945 tablet_bootstrap.cc:492] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: Bootstrap starting.
I20260925 01:30:07.265045 3949 tablet_bootstrap.cc:492] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: Bootstrap starting.
I20260925 01:30:07.266569 3945 tablet_bootstrap.cc:654] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:07.266831 3949 tablet_bootstrap.cc:654] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:07.268980 3945 tablet_bootstrap.cc:492] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: No bootstrap required, opened a new log
I20260925 01:30:07.269235 3945 ts_tablet_manager.cc:1397] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 01:30:07.270103 3945 raft_consensus.cc:348] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.270380 3945 raft_consensus.cc:374] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:07.270496 3945 raft_consensus.cc:729] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe60bb1ba7ef4cbeb3d885102b4b9805, State: Initialized, Role: FOLLOWER
I20260925 01:30:07.270773 3945 consensus_queue.cc:260] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.271330 3945 ts_tablet_manager.cc:1428] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 01:30:07.280172 3949 tablet_bootstrap.cc:492] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: No bootstrap required, opened a new log
I20260925 01:30:07.280401 3949 ts_tablet_manager.cc:1397] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20260925 01:30:07.281446 3949 raft_consensus.cc:348] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.281806 3949 raft_consensus.cc:374] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:07.281944 3949 raft_consensus.cc:729] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbedf691a54c958677b7f2015a6076, State: Initialized, Role: FOLLOWER
I20260925 01:30:07.282220 3949 consensus_queue.cc:260] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.283723 3949 ts_tablet_manager.cc:1428] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:07.468008 4003 raft_consensus.cc:482] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:30:07.468276 4003 raft_consensus.cc:504] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.469261 4003 leader_election.cc:288] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635), fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:07.469851 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec2167890614ab69e55cdd21909d642" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" is_pre_election: true
I20260925 01:30:07.469875 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec2167890614ab69e55cdd21909d642" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48fc46a2cc54054ab66353356ddb557" is_pre_election: true
I20260925 01:30:07.470119 3870 raft_consensus.cc:2454] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 0.
I20260925 01:30:07.470127 3590 raft_consensus.cc:2454] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 0.
I20260925 01:30:07.470623 3661 leader_election.cc:302] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fbedf691a54c958677b7f2015a6076, b48fc46a2cc54054ab66353356ddb557; no voters:
I20260925 01:30:07.470943 4003 raft_consensus.cc:2781] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:30:07.471091 4003 raft_consensus.cc:482] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:30:07.471189 4003 raft_consensus.cc:3037] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:07.474495 4003 raft_consensus.cc:504] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.475266 4003 leader_election.cc:288] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 election: Requested vote from peers b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635), fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:07.475943 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec2167890614ab69e55cdd21909d642" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b48fc46a2cc54054ab66353356ddb557"
I20260925 01:30:07.476008 3870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec2167890614ab69e55cdd21909d642" candidate_uuid: "30fbedf691a54c958677b7f2015a6076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805"
I20260925 01:30:07.476186 3590 raft_consensus.cc:3037] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:07.476257 3870 raft_consensus.cc:3037] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:07.478771 3870 raft_consensus.cc:2454] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 1.
I20260925 01:30:07.479297 3662 leader_election.cc:302] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fbedf691a54c958677b7f2015a6076, fe60bb1ba7ef4cbeb3d885102b4b9805; no voters:
I20260925 01:30:07.479355 3590 raft_consensus.cc:2454] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30fbedf691a54c958677b7f2015a6076 in term 1.
I20260925 01:30:07.479611 4003 raft_consensus.cc:2781] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:07.479864 4003 raft_consensus.cc:686] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 1 LEADER]: Becoming Leader. State: Replica: 30fbedf691a54c958677b7f2015a6076, State: Running, Role: LEADER
I20260925 01:30:07.480221 4003 consensus_queue.cc:237] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.483232 3432 catalog_manager.cc:5495] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30fbedf691a54c958677b7f2015a6076 (127.27.160.2). New cstate: current_term: 1 leader_uuid: "30fbedf691a54c958677b7f2015a6076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } health_report { overall_health: UNKNOWN } } }
I20260925 01:30:07.506898 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.501306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.27.160.2:38753)
0925 01:30:07.501550 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.501559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.501582 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:30:07.501900 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.501906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.501980 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.502093 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:07.502500 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.503312 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.504791 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.505386 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.505540 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.506050 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.506183 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.506300 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.506472 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.506565 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 01:30:07.509799 3590 raft_consensus.cc:1261] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Refusing update from remote peer 30fbedf691a54c958677b7f2015a6076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:07.509797 3870 raft_consensus.cc:1261] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Refusing update from remote peer 30fbedf691a54c958677b7f2015a6076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:07.510473 4003 consensus_queue.cc:1035] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:07.510762 3977 consensus_queue.cc:1035] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:07.526423 3432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:07.526721 3432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:07.530498 3432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: Sending DeleteTablet for 3 replicas of tablet 9ec2167890614ab69e55cdd21909d642
I20260925 01:30:07.531503 3571 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec2167890614ab69e55cdd21909d642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:43780
I20260925 01:30:07.531795 4007 tablet_replica.cc:330] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: stopping tablet replica
I20260925 01:30:07.531986 4007 raft_consensus.cc:2229] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:07.532209 4007 raft_consensus.cc:2258] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.533017 3850 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec2167890614ab69e55cdd21909d642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:38480
I20260925 01:30:07.533419 4009 tablet_replica.cc:330] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: stopping tablet replica
I20260925 01:30:07.534047 4009 raft_consensus.cc:2229] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:07.534380 4007 ts_tablet_manager.cc:1905] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.534509 4009 raft_consensus.cc:2258] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.536140 4009 ts_tablet_manager.cc:1905] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.541306 3850 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec2167890614ab69e55cdd21909d642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:38480
I20260925 01:30:07.541479 3709 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec2167890614ab69e55cdd21909d642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:54366
I20260925 01:30:07.541806 4008 tablet_replica.cc:330] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: stopping tablet replica
I20260925 01:30:07.542052 4008 raft_consensus.cc:2229] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:30:07.542431 4008 raft_consensus.cc:2258] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.542835 4007 ts_tablet_manager.cc:1918] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:07.543005 4007 log.cc:1196] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/9ec2167890614ab69e55cdd21909d642
I20260925 01:30:07.543067 4009 ts_tablet_manager.cc:1918] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:07.543221 4009 log.cc:1196] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/9ec2167890614ab69e55cdd21909d642
I20260925 01:30:07.543426 4007 ts_tablet_manager.cc:1939] T 9ec2167890614ab69e55cdd21909d642 P b48fc46a2cc54054ab66353356ddb557: Deleting consensus metadata
I20260925 01:30:07.543664 4009 ts_tablet_manager.cc:1939] T 9ec2167890614ab69e55cdd21909d642 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting consensus metadata
I20260925 01:30:07.543900 4008 ts_tablet_manager.cc:1905] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.544483 3418 catalog_manager.cc:4841] TS b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635): tablet 9ec2167890614ab69e55cdd21909d642 (table test-table [id=6195e171c5204b93bf913a7c77e7f66b]) successfully deleted
I20260925 01:30:07.544698 3419 catalog_manager.cc:4841] TS fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773): tablet 9ec2167890614ab69e55cdd21909d642 (table test-table [id=6195e171c5204b93bf913a7c77e7f66b]) successfully deleted
I20260925 01:30:07.544999 3432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:30:07.545711 3432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260925 01:30:07.545804 3419 catalog_manager.cc:4808] TS fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773): delete failed for tablet 9ec2167890614ab69e55cdd21909d642 because the tablet was not found. No further retry: Not found: Tablet not found: 9ec2167890614ab69e55cdd21909d642
I20260925 01:30:07.550890 4008 ts_tablet_manager.cc:1918] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:07.551177 4008 log.cc:1196] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/9ec2167890614ab69e55cdd21909d642
I20260925 01:30:07.551698 4008 ts_tablet_manager.cc:1939] T 9ec2167890614ab69e55cdd21909d642 P 30fbedf691a54c958677b7f2015a6076: Deleting consensus metadata
I20260925 01:30:07.552811 3420 catalog_manager.cc:4841] TS 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753): tablet 9ec2167890614ab69e55cdd21909d642 (table test-table [id=6195e171c5204b93bf913a7c77e7f66b]) successfully deleted
I20260925 01:30:07.553798 3709 tablet_service.cc:1467] Processing CreateTablet for tablet ea1cf9880fad421b8865d9b7c1582ae0 (DEFAULT_TABLE table=test-table [id=7b136a68b2a44de4ab97f86ddf87a2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:07.553786 3571 tablet_service.cc:1467] Processing CreateTablet for tablet ea1cf9880fad421b8865d9b7c1582ae0 (DEFAULT_TABLE table=test-table [id=7b136a68b2a44de4ab97f86ddf87a2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:07.554246 3709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1cf9880fad421b8865d9b7c1582ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:07.554245 3571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1cf9880fad421b8865d9b7c1582ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:07.557999 3850 tablet_service.cc:1467] Processing CreateTablet for tablet ea1cf9880fad421b8865d9b7c1582ae0 (DEFAULT_TABLE table=test-table [id=7b136a68b2a44de4ab97f86ddf87a2fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:07.558446 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1cf9880fad421b8865d9b7c1582ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:07.561061 3946 tablet_bootstrap.cc:492] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: Bootstrap starting.
I20260925 01:30:07.562626 3946 tablet_bootstrap.cc:654] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:07.564278 3949 tablet_bootstrap.cc:492] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: Bootstrap starting.
I20260925 01:30:07.565462 3945 tablet_bootstrap.cc:492] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: Bootstrap starting.
I20260925 01:30:07.566118 3949 tablet_bootstrap.cc:654] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:07.567199 3945 tablet_bootstrap.cc:654] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:07.567914 3946 tablet_bootstrap.cc:492] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: No bootstrap required, opened a new log
I20260925 01:30:07.568081 3946 ts_tablet_manager.cc:1397] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260925 01:30:07.569020 3946 raft_consensus.cc:348] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.569368 3946 raft_consensus.cc:374] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:07.569455 3946 raft_consensus.cc:729] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b48fc46a2cc54054ab66353356ddb557, State: Initialized, Role: FOLLOWER
I20260925 01:30:07.569780 3946 consensus_queue.cc:260] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.570410 3946 ts_tablet_manager.cc:1428] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 01:30:07.573230 3949 tablet_bootstrap.cc:492] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: No bootstrap required, opened a new log
I20260925 01:30:07.573478 3949 ts_tablet_manager.cc:1397] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260925 01:30:07.573825 3945 tablet_bootstrap.cc:492] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: No bootstrap required, opened a new log
I20260925 01:30:07.574008 3945 ts_tablet_manager.cc:1397] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260925 01:30:07.574558 3949 raft_consensus.cc:348] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.574731 3945 raft_consensus.cc:348] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.574976 3945 raft_consensus.cc:374] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:07.574988 3949 raft_consensus.cc:374] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:07.575067 3945 raft_consensus.cc:729] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe60bb1ba7ef4cbeb3d885102b4b9805, State: Initialized, Role: FOLLOWER
I20260925 01:30:07.575203 3949 raft_consensus.cc:729] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30fbedf691a54c958677b7f2015a6076, State: Initialized, Role: FOLLOWER
I20260925 01:30:07.575366 3945 consensus_queue.cc:260] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.575551 3949 consensus_queue.cc:260] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.576117 3945 ts_tablet_manager.cc:1428] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 01:30:07.577315 3949 ts_tablet_manager.cc:1428] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 01:30:07.591034 3956 raft_consensus.cc:482] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:30:07.591292 3956 raft_consensus.cc:504] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.591969 3956 leader_election.cc:288] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753), fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:07.592520 3730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1cf9880fad421b8865d9b7c1582ae0" candidate_uuid: "b48fc46a2cc54054ab66353356ddb557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fbedf691a54c958677b7f2015a6076" is_pre_election: true
I20260925 01:30:07.592551 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1cf9880fad421b8865d9b7c1582ae0" candidate_uuid: "b48fc46a2cc54054ab66353356ddb557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" is_pre_election: true
I20260925 01:30:07.592844 3730 raft_consensus.cc:2454] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b48fc46a2cc54054ab66353356ddb557 in term 0.
I20260925 01:30:07.592837 3869 raft_consensus.cc:2454] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b48fc46a2cc54054ab66353356ddb557 in term 0.
I20260925 01:30:07.593353 3522 leader_election.cc:302] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b48fc46a2cc54054ab66353356ddb557, fe60bb1ba7ef4cbeb3d885102b4b9805; no voters:
I20260925 01:30:07.593740 3956 raft_consensus.cc:2781] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:30:07.593852 3956 raft_consensus.cc:482] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:30:07.593989 3956 raft_consensus.cc:3037] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:07.596108 3956 raft_consensus.cc:504] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.596688 3956 leader_election.cc:288] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [CANDIDATE]: Term 1 election: Requested vote from peers 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753), fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773)
I20260925 01:30:07.597245 3730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1cf9880fad421b8865d9b7c1582ae0" candidate_uuid: "b48fc46a2cc54054ab66353356ddb557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30fbedf691a54c958677b7f2015a6076"
I20260925 01:30:07.597290 3869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1cf9880fad421b8865d9b7c1582ae0" candidate_uuid: "b48fc46a2cc54054ab66353356ddb557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805"
I20260925 01:30:07.597498 3730 raft_consensus.cc:3037] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:07.597500 3869 raft_consensus.cc:3037] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:07.599725 3730 raft_consensus.cc:2454] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48fc46a2cc54054ab66353356ddb557 in term 1.
I20260925 01:30:07.600266 3523 leader_election.cc:302] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30fbedf691a54c958677b7f2015a6076, b48fc46a2cc54054ab66353356ddb557; no voters:
I20260925 01:30:07.600414 3869 raft_consensus.cc:2454] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b48fc46a2cc54054ab66353356ddb557 in term 1.
I20260925 01:30:07.600593 3956 raft_consensus.cc:2781] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:07.601094 3956 raft_consensus.cc:686] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 1 LEADER]: Becoming Leader. State: Replica: b48fc46a2cc54054ab66353356ddb557, State: Running, Role: LEADER
I20260925 01:30:07.602275 3956 consensus_queue.cc:237] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } }
I20260925 01:30:07.605194 3434 catalog_manager.cc:5495] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 reported cstate change: term changed from 0 to 1, leader changed from <none> to b48fc46a2cc54054ab66353356ddb557 (127.27.160.1). New cstate: current_term: 1 leader_uuid: "b48fc46a2cc54054ab66353356ddb557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b48fc46a2cc54054ab66353356ddb557" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 41635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 } health_report { overall_health: UNKNOWN } } }
I20260925 01:30:07.633002 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.627369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.27.160.1:41635)
0925 01:30:07.627851 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.627857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.627875 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:30:07.627943 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.627950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.628027 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.628134 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:07.628487 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.629619 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.630890 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.631546 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.631670 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.632194 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.632331 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.632424 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.632558 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.632646 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":231,"threads_started":1}
I20260925 01:30:07.648999 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.642627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:32973 (local address 127.27.160.2:38753)
0925 01:30:07.642870 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.642880 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.643452 (+ 572us) server_negotiation.cc:407] Connection header received
0925 01:30:07.643665 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.643668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.643735 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.643815 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:07.644349 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.645083 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.647133 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.647621 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.647718 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.648269 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.648375 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.648477 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.648660 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.648731 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:30:07.649161 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.642628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:38753 (local address 127.27.160.1:32973)
0925 01:30:07.643332 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:07.643360 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:07.643514 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:07.643852 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:07.643857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:07.643871 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:30:07.644203 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:07.644216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.645257 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.645263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:07.646998 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:07.647015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.647811 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.647818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:07.648002 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.648095 (+ 93us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.648478 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.648501 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:30:07.648596 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 01:30:07.648745 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":560,"threads_started":1}
I20260925 01:30:07.658475 3730 raft_consensus.cc:1261] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Refusing update from remote peer b48fc46a2cc54054ab66353356ddb557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:07.658496 3869 raft_consensus.cc:1261] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Refusing update from remote peer b48fc46a2cc54054ab66353356ddb557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:07.659049 3956 consensus_queue.cc:1035] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe60bb1ba7ef4cbeb3d885102b4b9805" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:07.659346 4032 consensus_queue.cc:1035] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30fbedf691a54c958677b7f2015a6076" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 38753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:07.692974 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.686240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:36855 (local address 127.27.160.1:41635)
0925 01:30:07.686409 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.686416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.686497 (+ 81us) server_negotiation.cc:407] Connection header received
0925 01:30:07.686752 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.686758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.686860 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.686925 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:07.687598 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.688661 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.691036 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.691644 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.691764 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.692179 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.692298 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.692380 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.692631 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.692709 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:30:07.693178 4042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.685769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:41635 (local address 127.27.160.2:36855)
0925 01:30:07.686390 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:07.686414 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:07.686558 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:07.687149 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:07.687154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:07.687167 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:30:07.687441 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:07.687450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.688788 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.688795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:07.690758 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:07.690773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.691826 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.691831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:07.691967 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.692005 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:30:07.692388 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:30:07.692397 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:30:07.692596 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 01:30:07.692729 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":845,"threads_started":1}
I20260925 01:30:07.724007 3434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:07.724236 3434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:07.727116 3434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 524485c44abb4e0fa8527ae9cca62e89: Sending DeleteTablet for 3 replicas of tablet ea1cf9880fad421b8865d9b7c1582ae0
I20260925 01:30:07.727857 3571 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1cf9880fad421b8865d9b7c1582ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:43780
I20260925 01:30:07.728101 3709 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1cf9880fad421b8865d9b7c1582ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:54366
I20260925 01:30:07.728089 3850 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1cf9880fad421b8865d9b7c1582ae0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:07 UTC) from {username='oryx', principal='oryx/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:38480
I20260925 01:30:07.728154 4007 tablet_replica.cc:330] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: stopping tablet replica
I20260925 01:30:07.728331 4007 raft_consensus.cc:2229] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:30:07.728394 4009 tablet_replica.cc:330] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: stopping tablet replica
I20260925 01:30:07.728400 4008 tablet_replica.cc:330] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: stopping tablet replica
I20260925 01:30:07.728564 4009 raft_consensus.cc:2229] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:07.728585 4008 raft_consensus.cc:2229] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:07.728682 4007 raft_consensus.cc:2258] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.728766 4009 raft_consensus.cc:2258] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.728798 4008 raft_consensus.cc:2258] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:07.730057 4007 ts_tablet_manager.cc:1905] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.730057 4008 ts_tablet_manager.cc:1905] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.732324 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 3505
I20260925 01:30:07.732879 4009 ts_tablet_manager.cc:1905] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:07.737288 4008 ts_tablet_manager.cc:1918] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:30:07.737448 4008 log.cc:1196] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/ea1cf9880fad421b8865d9b7c1582ae0
I20260925 01:30:07.737919 4008 ts_tablet_manager.cc:1939] T ea1cf9880fad421b8865d9b7c1582ae0 P 30fbedf691a54c958677b7f2015a6076: Deleting consensus metadata
I20260925 01:30:07.739156 3420 catalog_manager.cc:4841] TS 30fbedf691a54c958677b7f2015a6076 (127.27.160.2:38753): tablet ea1cf9880fad421b8865d9b7c1582ae0 (table test-table [id=7b136a68b2a44de4ab97f86ddf87a2fb]) successfully deleted
I20260925 01:30:07.741642 4007 ts_tablet_manager.cc:1918] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:30:07.741796 4007 log.cc:1196] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/ea1cf9880fad421b8865d9b7c1582ae0
I20260925 01:30:07.742156 4007 ts_tablet_manager.cc:1939] T ea1cf9880fad421b8865d9b7c1582ae0 P b48fc46a2cc54054ab66353356ddb557: Deleting consensus metadata
I20260925 01:30:07.743171 3418 catalog_manager.cc:4841] TS b48fc46a2cc54054ab66353356ddb557 (127.27.160.1:41635): tablet ea1cf9880fad421b8865d9b7c1582ae0 (table test-table [id=7b136a68b2a44de4ab97f86ddf87a2fb]) successfully deleted
I20260925 01:30:07.745018 4009 ts_tablet_manager.cc:1918] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:30:07.745208 4009 log.cc:1196] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/ea1cf9880fad421b8865d9b7c1582ae0
I20260925 01:30:07.745579 4009 ts_tablet_manager.cc:1939] T ea1cf9880fad421b8865d9b7c1582ae0 P fe60bb1ba7ef4cbeb3d885102b4b9805: Deleting consensus metadata
I20260925 01:30:07.746766 3419 catalog_manager.cc:4841] TS fe60bb1ba7ef4cbeb3d885102b4b9805 (127.27.160.3:44773): tablet ea1cf9880fad421b8865d9b7c1582ae0 (table test-table [id=7b136a68b2a44de4ab97f86ddf87a2fb]) successfully deleted
Sep 25 01:30:07 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.1@KRBTEST.COM
I20260925 01:30:07.759333 4034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.733722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.27.160.1:41635)
0925 01:30:07.733895 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.733902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.743299 (+ 9397us) server_negotiation.cc:407] Connection header received
0925 01:30:07.743794 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.743800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.743901 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.744006 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:07.745792 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.746830 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.748112 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.748779 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.748908 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.753360 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:07.753377 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:07.753398 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:07.753439 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:07.756368 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:07.757273 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:07.757281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:07.757283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:07.757425 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:07.757853 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:07.757857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:07.757859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:07.758205 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:07.758427 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.758837 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.759001 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:30:07.760219 3636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:07.887557 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 3505
I20260925 01:30:07.919503 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 3645
Sep 25 01:30:07 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.2@KRBTEST.COM
I20260925 01:30:07.938069 3993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:07.920438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.27.160.2:38753)
0925 01:30:07.920609 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:07.920613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:07.920629 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:30:07.920751 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:07.920755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:07.920817 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:07.920905 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:07.922593 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.923336 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.925318 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:07.925775 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:07.925955 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:07.930809 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:07.930828 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:07.930849 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:07.930895 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:07.934827 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:07.935692 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:07.935702 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:07.935706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:07.935866 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:07.936287 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:07.936292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:07.936296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:07.936793 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:07.937111 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:07.937447 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 01:30:07.937702 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:30:07.938814 3775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:08.100174 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 3645
I20260925 01:30:08.126617 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 3784
Sep 25 01:30:08 dist-test-slave-h6hf krb5kdc[3389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.160.3@KRBTEST.COM
I20260925 01:30:08.144094 4047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:08.127527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.27.160.3:44773)
0925 01:30:08.128028 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:08.128032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:08.128048 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:30:08.128100 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:08.128104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:08.128162 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:08.128251 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:08.129730 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:08.130605 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:08.131853 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:08.132395 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:08.132497 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:08.137044 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:08.137064 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:08.137082 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:08.137221 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:08.141036 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:08.141959 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:08.141968 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:08.141970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:08.142131 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:08.142522 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:08.142527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:08.142529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:08.143063 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:08.143348 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:08.143566 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 01:30:08.143779 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":257,"threads_started":1}
I20260925 01:30:08.144906 3915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:08.250308 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 3784
I20260925 01:30:08.268411 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 3402
I20260925 01:30:08.269637 3474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:08.386402 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 3402
2026-09-25T01:30:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7030 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:30:08 dist-test-slave-h6hf krb5kdc[4052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:30:08 dist-test-slave-h6hf krb5kdc[4052](info): set up 2 sockets
Sep 25 01:30:08 dist-test-slave-h6hf krb5kdc[4052](info): commencing operation
krb5kdc: starting...
W20260925 01:30:09.491089 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:30:09 dist-test-slave-h6hf krb5kdc[4052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:30:09.728394 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:33829
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:35085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:33829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:10.059919 4065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:10.060285 4065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:10.060467 4065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:10.067696 4065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:30:10.067860 4065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:10.067973 4065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:10.068081 4065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:30:10.068153 4065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:30:10.078236 4065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:35085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:33829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:10.079638 4065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:10.080989 4065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:10.093262 4070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:10.093174 4073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:10.093204 4071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:10.093592 4065 server_base.cc:1034] running on GCE node
I20260925 01:30:10.094679 4065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:10.096524 4065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:10.097760 4065 hybrid_clock.cc:648] HybridClock initialized: now 1790299810097696 us; error 58 us; skew 500 ppm
I20260925 01:30:10.098129 4065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T01:30:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1779 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:30:10 dist-test-slave-h6hf krb5kdc[4079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:30:10 dist-test-slave-h6hf krb5kdc[4079](info): set up 2 sockets
Sep 25 01:30:10 dist-test-slave-h6hf krb5kdc[4079](info): commencing operation
krb5kdc: starting...
W20260925 01:30:11.327060 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:30:11 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:30:11.558888 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44219
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:11.894169 4092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:11.894753 4092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:11.894980 4092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:11.903349 4092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:30:11.903538 4092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:11.903666 4092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:11.903782 4092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:30:11.903884 4092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:30:11.915220 4092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:11.916761 4092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:11.918236 4092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:11.931162 4097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:11.931126 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:11.931159 4098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:11.932068 4092 server_base.cc:1034] running on GCE node
I20260925 01:30:11.932819 4092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:11.934329 4092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:11.935575 4092 hybrid_clock.cc:648] HybridClock initialized: now 1790299811935520 us; error 52 us; skew 500 ppm
I20260925 01:30:11.935912 4092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:11 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:11.943014 4092 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:30:11.944736 4092 webserver.cc:460] Webserver started at http://127.27.160.62:46415/ using document root <none> and password file <none>
I20260925 01:30:11.945566 4092 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:11.945677 4092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:11.945959 4092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:11.949051 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "6634b23a1e9948f589cd033f3df7380b"
format_stamp: "Formatted at 2026-09-25 01:30:11 on dist-test-slave-h6hf"
server_key: "c60be4da1e3ae5946bd202bedbba3e3a"
server_key_iv: "93c51fd62123e93934f8b76fb3f41c2c"
server_key_version: "encryptionkey@0"
I20260925 01:30:11.949958 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "6634b23a1e9948f589cd033f3df7380b"
format_stamp: "Formatted at 2026-09-25 01:30:11 on dist-test-slave-h6hf"
server_key: "c60be4da1e3ae5946bd202bedbba3e3a"
server_key_iv: "93c51fd62123e93934f8b76fb3f41c2c"
server_key_version: "encryptionkey@0"
I20260925 01:30:11.955528 4092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:30:11.959539 4107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:11.961097 4092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:30:11.961339 4092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "6634b23a1e9948f589cd033f3df7380b"
format_stamp: "Formatted at 2026-09-25 01:30:11 on dist-test-slave-h6hf"
server_key: "c60be4da1e3ae5946bd202bedbba3e3a"
server_key_iv: "93c51fd62123e93934f8b76fb3f41c2c"
server_key_version: "encryptionkey@0"
I20260925 01:30:11.961537 4092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:11.980489 4092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:11.984984 4092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:11.985344 4092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:12.006126 4092 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:44219
I20260925 01:30:12.006145 4158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:44219 every 8 connection(s)
I20260925 01:30:12.007827 4092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:30:12.011283 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 4092
I20260925 01:30:12.011539 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:30:12.011984 4159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:12.012255 28288 external_mini_cluster.cc:1428] Setting key ec21cef03410cfbe41f82894f1901410
I20260925 01:30:12.027050 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Bootstrap starting.
Sep 25 01:30:12 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:30:12.031235 4159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:12.032676 4159 log.cc:824] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:12.035933 4159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: No bootstrap required, opened a new log
I20260925 01:30:12.037055 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:12.014375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36830 (local address 127.27.160.62:44219)
0925 01:30:12.015287 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:12.015305 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:12.015363 (+ 58us) server_negotiation.cc:407] Connection header received
0925 01:30:12.017286 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:12.017319 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:12.017910 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:12.018400 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:12.019776 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:12.021772 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:12.023113 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:12.024274 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:12.024481 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:12.028520 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:12.028535 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:12.028574 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:12.028639 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:12.032717 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:12.033590 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:12.033598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:12.033608 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:12.033811 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:12.034236 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:12.034239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:12.034241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:12.034772 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:12.035088 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:12.035545 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 01:30:12.035983 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":356,"threads_started":1}
I20260925 01:30:12.044163 4159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } }
I20260925 01:30:12.044521 4159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:12.044649 4159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6634b23a1e9948f589cd033f3df7380b, State: Initialized, Role: FOLLOWER
I20260925 01:30:12.045343 4159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } }
I20260925 01:30:12.045670 4159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:30:12.045805 4159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:30:12.045979 4159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:12.048151 4159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } }
I20260925 01:30:12.048666 4159 leader_election.cc:302] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6634b23a1e9948f589cd033f3df7380b; no voters:
I20260925 01:30:12.049307 4159 leader_election.cc:288] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:30:12.049431 4164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:12.050189 4164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 1 LEADER]: Becoming Leader. State: Replica: 6634b23a1e9948f589cd033f3df7380b, State: Running, Role: LEADER
I20260925 01:30:12.050694 4164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } }
I20260925 01:30:12.051581 4159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:30:12.053679 4166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6634b23a1e9948f589cd033f3df7380b. Latest consensus state: current_term: 1 leader_uuid: "6634b23a1e9948f589cd033f3df7380b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } } }
I20260925 01:30:12.054109 4166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:12.053937 4165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6634b23a1e9948f589cd033f3df7380b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } } }
I20260925 01:30:12.054245 4165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:12.058439 4171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:30:12.065169 4171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:30:12.076749 4171 catalog_manager.cc:1347] Generated new cluster ID: f1209f9319384d4883e35dfd0256a108
I20260925 01:30:12.076879 4171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:30:12.090178 4171 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:30:12.091531 4171 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:30:12.101972 4171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Generated new TSK 0
I20260925 01:30:12.102850 4171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:30:12.165609 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44219
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:12.573824 4185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:12.574172 4185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:12.574352 4185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:12.582700 4185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:12.582895 4185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:12.583127 4185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:30:12.594584 4185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:12.596073 4185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:12.597589 4185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:12.611222 4191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:12.611363 4190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:12.611708 4193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:12.612212 4185 server_base.cc:1034] running on GCE node
I20260925 01:30:12.612769 4185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:12.613759 4185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:12.614964 4185 hybrid_clock.cc:648] HybridClock initialized: now 1790299812614920 us; error 33 us; skew 500 ppm
I20260925 01:30:12.615242 4185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:12 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299812, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:12.618373 4185 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:30:12.619861 4185 webserver.cc:460] Webserver started at http://127.27.160.1:38607/ using document root <none> and password file <none>
I20260925 01:30:12.620589 4185 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:12.620739 4185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:12.621066 4185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:12.624027 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "be5f6c22b33a480f99cd00896778725d"
format_stamp: "Formatted at 2026-09-25 01:30:12 on dist-test-slave-h6hf"
server_key: "2240ac57ee61c22e0dc56c7f402c0e8b"
server_key_iv: "f82918db4b591fd9a3ac24cbfb529992"
server_key_version: "encryptionkey@0"
I20260925 01:30:12.624841 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "be5f6c22b33a480f99cd00896778725d"
format_stamp: "Formatted at 2026-09-25 01:30:12 on dist-test-slave-h6hf"
server_key: "2240ac57ee61c22e0dc56c7f402c0e8b"
server_key_iv: "f82918db4b591fd9a3ac24cbfb529992"
server_key_version: "encryptionkey@0"
I20260925 01:30:12.630126 4185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:30:12.633730 4200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:12.635085 4185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:30:12.635269 4185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "be5f6c22b33a480f99cd00896778725d"
format_stamp: "Formatted at 2026-09-25 01:30:12 on dist-test-slave-h6hf"
server_key: "2240ac57ee61c22e0dc56c7f402c0e8b"
server_key_iv: "f82918db4b591fd9a3ac24cbfb529992"
server_key_version: "encryptionkey@0"
I20260925 01:30:12.635478 4185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:12.671015 4185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:12.675370 4185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:12.675690 4185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:12.676678 4185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:12.678128 4185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:12.678233 4185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:12.678390 4185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:12.678462 4185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:12.712211 4185 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:42227
I20260925 01:30:12.712261 4312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:42227 every 8 connection(s)
I20260925 01:30:12.713588 4185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:30:12.722962 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 4185
I20260925 01:30:12.723192 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:30:12.724017 28288 external_mini_cluster.cc:1428] Setting key 086a867dc44be80427ef46556a0624a1
Sep 25 01:30:12 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299812, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:30:12.733898 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:12.716252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:51825 (local address 127.27.160.62:44219)
0925 01:30:12.716683 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:12.716688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:12.717685 (+ 997us) server_negotiation.cc:407] Connection header received
0925 01:30:12.719452 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:12.719457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:12.719523 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:12.719622 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:12.721066 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:12.721848 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:12.723512 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:12.724193 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:12.724331 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:12.728115 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:12.728120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:12.728124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:12.728154 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:12.730731 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:12.731486 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:12.731494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:12.731500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:12.731633 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:12.732062 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:12.732070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:12.732075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:12.732354 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:12.732509 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:12.733247 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 01:30:12.733476 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":215,"threads_started":1}
I20260925 01:30:12.734937 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:12.716775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:51825)
0925 01:30:12.717526 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:12.717570 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:12.719201 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:12.719832 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:12.719840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:12.720360 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:12.720903 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:12.720921 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:12.721986 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:12.721989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:12.723364 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:12.723376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:12.724404 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:12.724408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:12.724608 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:12.725357 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:12.725385 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:12.727895 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:12.730893 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:12.730901 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:12.730920 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:12.731364 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:12.731746 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:12.731751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:12.731755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:12.731952 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:12.732483 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:12.732491 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:12.732856 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 01:30:12.733163 (+ 307us) client_negotiation.cc:240] Negotiation successful
0925 01:30:12.733728 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":330,"threads_started":1}
I20260925 01:30:12.736689 4313 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44219
I20260925 01:30:12.736994 4313 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:12.737746 4313 heartbeater.cc:511] Master 127.27.160.62:44219 requested a full tablet report, sending...
I20260925 01:30:12.740087 4124 ts_manager.cc:194] Registered new tserver with Master: be5f6c22b33a480f99cd00896778725d (127.27.160.1:42227)
I20260925 01:30:12.742385 4124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:51825
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:30:12.776190 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44219
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:13.102761 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:13.103050 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:13.103145 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:13.110193 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:13.110302 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:13.110414 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:30:13.120103 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:13.121464 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:13.122732 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:13.136024 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:13.136090 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:13.136133 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:13.136693 4319 server_base.cc:1034] running on GCE node
I20260925 01:30:13.137437 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:13.138434 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:13.139631 4319 hybrid_clock.cc:648] HybridClock initialized: now 1790299813139576 us; error 54 us; skew 500 ppm
I20260925 01:30:13.139951 4319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:13 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299813, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:13.143255 4319 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:30:13.144830 4319 webserver.cc:460] Webserver started at http://127.27.160.2:38561/ using document root <none> and password file <none>
I20260925 01:30:13.145651 4319 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:13.145792 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:13.146128 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:13.149001 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "30a2e77002d64dc5840b80e11fe3595b"
format_stamp: "Formatted at 2026-09-25 01:30:13 on dist-test-slave-h6hf"
server_key: "a2bfa025589379f1d4a8ae0fc000e9ff"
server_key_iv: "49b17c553dcd9423c9a7af996e820048"
server_key_version: "encryptionkey@0"
I20260925 01:30:13.149842 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "30a2e77002d64dc5840b80e11fe3595b"
format_stamp: "Formatted at 2026-09-25 01:30:13 on dist-test-slave-h6hf"
server_key: "a2bfa025589379f1d4a8ae0fc000e9ff"
server_key_iv: "49b17c553dcd9423c9a7af996e820048"
server_key_version: "encryptionkey@0"
I20260925 01:30:13.154950 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:30:13.158418 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:13.159693 4319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:30:13.159865 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "30a2e77002d64dc5840b80e11fe3595b"
format_stamp: "Formatted at 2026-09-25 01:30:13 on dist-test-slave-h6hf"
server_key: "a2bfa025589379f1d4a8ae0fc000e9ff"
server_key_iv: "49b17c553dcd9423c9a7af996e820048"
server_key_version: "encryptionkey@0"
I20260925 01:30:13.160073 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:13.177093 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:13.181924 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:13.182252 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:13.183284 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:13.184705 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:13.184813 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:13.184958 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:13.185029 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:13.218938 4319 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:42939
I20260925 01:30:13.218978 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:42939 every 8 connection(s)
I20260925 01:30:13.220218 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:30:13.227320 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 4319
I20260925 01:30:13.227492 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:30:13.227943 28288 external_mini_cluster.cc:1428] Setting key 88958a0f72b953dbfe828425ea2ac3d5
Sep 25 01:30:13 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299813, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:30:13.241026 4315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:13.222782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:46407 (local address 127.27.160.62:44219)
0925 01:30:13.222975 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:13.222979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:13.224100 (+ 1121us) server_negotiation.cc:407] Connection header received
0925 01:30:13.225990 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:13.225996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:13.226098 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:13.226209 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:13.227798 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.228617 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.230445 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.230877 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.231016 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:13.234500 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:13.234505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:13.234509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:13.234540 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:13.237831 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:13.238576 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:13.238582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:13.238586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:13.238724 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:13.239186 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:13.239190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:13.239196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:13.239528 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:13.239681 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:13.240430 (+ 749us) server_negotiation.cc:299] Negotiation successful
0925 01:30:13.240659 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:30:13.242075 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:13.223232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:46407)
0925 01:30:13.223929 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:13.223968 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:13.225705 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:13.226454 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:13.226463 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:13.226957 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:13.227630 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:13.227650 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.228793 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.228799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:13.230286 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:13.230298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.231124 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.231132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:13.231365 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:13.232078 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:13.232099 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:13.234272 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:13.237982 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:13.237999 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:13.238015 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:13.238432 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:13.238840 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:13.238845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:13.238848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:13.239064 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:13.239657 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:13.239665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:13.240074 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 01:30:13.240341 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 01:30:13.240917 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":267,"threads_started":1}
I20260925 01:30:13.243585 4447 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44219
I20260925 01:30:13.243925 4447 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:13.244668 4447 heartbeater.cc:511] Master 127.27.160.62:44219 requested a full tablet report, sending...
I20260925 01:30:13.246318 4124 ts_manager.cc:194] Registered new tserver with Master: 30a2e77002d64dc5840b80e11fe3595b (127.27.160.2:42939)
I20260925 01:30:13.247651 4124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:46407
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:30:13.281049 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44219
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:13.612964 4452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:13.613341 4452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:13.613447 4452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:13.620764 4452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:13.620900 4452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:13.621016 4452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:30:13.631487 4452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:13.632782 4452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:13.634161 4452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:13.647847 4460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:13.647909 4457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:13.647924 4458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:13.648737 4452 server_base.cc:1034] running on GCE node
I20260925 01:30:13.649497 4452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:13.650609 4452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:13.651844 4452 hybrid_clock.cc:648] HybridClock initialized: now 1790299813651765 us; error 74 us; skew 500 ppm
I20260925 01:30:13.652247 4452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:13 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299813, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:13.655576 4452 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:30:13.657032 4452 webserver.cc:460] Webserver started at http://127.27.160.3:32863/ using document root <none> and password file <none>
I20260925 01:30:13.657903 4452 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:13.658015 4452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:13.658286 4452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:13.661403 4452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "39a6f8779e3c4603901c03abe86bf586"
format_stamp: "Formatted at 2026-09-25 01:30:13 on dist-test-slave-h6hf"
server_key: "c6046cefc4b36269987583903b866c0f"
server_key_iv: "3ca0907c076b02d1ec04e0d86362fc37"
server_key_version: "encryptionkey@0"
I20260925 01:30:13.662351 4452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "39a6f8779e3c4603901c03abe86bf586"
format_stamp: "Formatted at 2026-09-25 01:30:13 on dist-test-slave-h6hf"
server_key: "c6046cefc4b36269987583903b866c0f"
server_key_iv: "3ca0907c076b02d1ec04e0d86362fc37"
server_key_version: "encryptionkey@0"
I20260925 01:30:13.669061 4452 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 01:30:13.673210 4467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:13.674775 4452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:13.675058 4452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "39a6f8779e3c4603901c03abe86bf586"
format_stamp: "Formatted at 2026-09-25 01:30:13 on dist-test-slave-h6hf"
server_key: "c6046cefc4b36269987583903b866c0f"
server_key_iv: "3ca0907c076b02d1ec04e0d86362fc37"
server_key_version: "encryptionkey@0"
I20260925 01:30:13.675338 4452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:13.695829 4452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:13.700572 4452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:13.700893 4452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:13.701936 4452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:13.703397 4452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:13.703473 4452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:13.703645 4452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:13.703699 4452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:13.738161 4452 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:40303
I20260925 01:30:13.738201 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:40303 every 8 connection(s)
I20260925 01:30:13.739584 4452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:30:13.743978 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 4452
I20260925 01:30:13.744187 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:30:13.744629 28288 external_mini_cluster.cc:1428] Setting key ec2e46c5ee994843b25fa9ba11ac4625
I20260925 01:30:13.746315 4313 heartbeater.cc:503] Master 127.27.160.62:44219 was elected leader, sending a full tablet report...
Sep 25 01:30:13 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299813, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:30:13.760243 4582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:13.742259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:57123 (local address 127.27.160.62:44219)
0925 01:30:13.742648 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:13.742653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:13.743689 (+ 1036us) server_negotiation.cc:407] Connection header received
0925 01:30:13.745931 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:13.745936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:13.746025 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:13.746156 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:13.748218 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.748963 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.750519 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.750909 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.751044 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:13.754562 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:13.754567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:13.754570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:13.754603 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:13.757183 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:13.757898 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:13.757903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:13.757907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:13.758033 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:13.758467 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:13.758472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:13.758475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:13.758751 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:13.758907 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:13.759615 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 01:30:13.759836 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":188,"threads_started":1}
I20260925 01:30:13.761475 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:13.742834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:57123)
0925 01:30:13.743537 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:13.743573 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:13.745615 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:13.746414 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:13.746427 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:13.747254 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:13.748057 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:13.748077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.749105 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.749111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:13.750354 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:13.750366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.751133 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.751140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:13.751389 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:13.752062 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:13.752088 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:13.754333 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:13.757320 (+ 2987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:13.757328 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:13.757345 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:13.757765 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:13.758142 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:13.758147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:13.758151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:13.758347 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:13.758879 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:13.758887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:13.759240 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 01:30:13.759510 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 01:30:13.760072 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":279,"threads_started":1}
I20260925 01:30:13.762787 4580 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44219
I20260925 01:30:13.763115 4580 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:13.763955 4580 heartbeater.cc:511] Master 127.27.160.62:44219 requested a full tablet report, sending...
I20260925 01:30:13.765466 4123 ts_manager.cc:194] Registered new tserver with Master: 39a6f8779e3c4603901c03abe86bf586 (127.27.160.3:40303)
I20260925 01:30:13.766347 4123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:57123
I20260925 01:30:13.773015 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab.
Entry for principal oryx/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab.
WARNING: no policy specified for oryx/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab.
Entry for principal oryx/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab.
WARNING: no policy specified for oryx/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab.
Entry for principal oryx/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab.
I20260925 01:30:13.919570 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4185
Sep 25 01:30:13 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:30:13.936282 4591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:13.920733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.27.160.1:42227)
0925 01:30:13.921633 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:13.921640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:13.921668 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:30:13.921730 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:13.921736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:13.921965 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:13.922106 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:13.923458 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.924504 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.925843 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:13.926421 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:13.926518 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:13.930605 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:13.930620 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:13.930635 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:13.930678 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:13.933619 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:13.934316 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:13.934323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:13.934325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:13.934451 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:13.934799 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:13.934802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:13.934804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:13.935151 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:13.935373 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:13.935654 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 01:30:13.935857 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":557,"threads_started":1}
I20260925 01:30:13.937582 4308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:14.021160 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 4185
I20260925 01:30:14.038478 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4319
Sep 25 01:30:14 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:30:14.054813 4593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:14.039421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.27.160.2:42939)
0925 01:30:14.039989 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:14.040004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:14.040029 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:30:14.040096 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:14.040103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:14.040313 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:14.040511 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:14.041767 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:14.043001 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:14.044297 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:14.044848 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:14.044944 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:14.049402 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:14.049414 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:14.049429 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:14.049464 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:14.052156 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:14.052859 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:14.052866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:14.052868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:14.052986 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:14.053360 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:14.053363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:14.053365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:14.053713 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:14.053936 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:14.054234 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 01:30:14.054462 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
I20260925 01:30:14.056133 4442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:14.140252 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 4319
I20260925 01:30:14.156852 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4452
Sep 25 01:30:14 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:30:14.173815 4595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:14.157933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.27.160.3:40303)
0925 01:30:14.158662 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:14.158668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:14.158698 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:30:14.158762 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:14.158770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:14.159002 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:14.159139 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:14.160393 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:14.161577 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:14.162959 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:14.163509 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:14.163604 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:14.168140 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:14.168151 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:14.168167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:14.168204 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:14.171024 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:14.171685 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:14.171692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:14.171694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:14.171824 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:14.172218 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:14.172223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:14.172226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:14.172652 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:14.172905 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:14.173248 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 01:30:14.173460 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":423,"threads_started":1}
I20260925 01:30:14.175208 4575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:14.261307 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 4452
I20260925 01:30:14.277105 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4092
I20260925 01:30:14.278383 4154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:14.382155 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 4092
I20260925 01:30:14.403205 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44219
--webserver_interface=127.27.160.62
--webserver_port=46415
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:14.743052 4598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:14.743355 4598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:14.743515 4598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:14.751747 4598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:30:14.751866 4598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:14.751950 4598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:14.752032 4598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:30:14.752117 4598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:30:14.762132 4598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:14.763554 4598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:14.764952 4598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:14.777033 4603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:14.777097 4606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:14.777185 4604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:14.777871 4598 server_base.cc:1034] running on GCE node
I20260925 01:30:14.778682 4598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:14.780189 4598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:14.781428 4598 hybrid_clock.cc:648] HybridClock initialized: now 1790299814781380 us; error 44 us; skew 500 ppm
I20260925 01:30:14.781782 4598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:14 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299814, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:14.784945 4598 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:30:14.786492 4598 webserver.cc:460] Webserver started at http://127.27.160.62:46415/ using document root <none> and password file <none>
I20260925 01:30:14.787211 4598 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:14.787380 4598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:14.792924 4598 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:30:14.795950 4613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:14.797454 4598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:14.797643 4598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "6634b23a1e9948f589cd033f3df7380b"
format_stamp: "Formatted at 2026-09-25 01:30:11 on dist-test-slave-h6hf"
server_key: "c60be4da1e3ae5946bd202bedbba3e3a"
server_key_iv: "93c51fd62123e93934f8b76fb3f41c2c"
server_key_version: "encryptionkey@0"
I20260925 01:30:14.798666 4598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:14.826143 4598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:14.830295 4598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:14.830582 4598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:14.850193 4598 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:44219
I20260925 01:30:14.850214 4664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:44219 every 8 connection(s)
I20260925 01:30:14.851786 4598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:30:14.857017 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 4598
I20260925 01:30:14.858152 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:42227
--local_ip_for_outbound_sockets=127.27.160.1
--tserver_master_addrs=127.27.160.62:44219
--webserver_port=38607
--webserver_interface=127.27.160.1
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.160.1
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:30:14.860486 4665 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:30:14.866384 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Bootstrap starting.
I20260925 01:30:14.887645 4665 log.cc:824] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:14.897521 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:30:14.898113 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Bootstrap complete.
I20260925 01:30:14.908771 4665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } }
I20260925 01:30:14.909801 4665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6634b23a1e9948f589cd033f3df7380b, State: Initialized, Role: FOLLOWER
I20260925 01:30:14.910565 4665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } }
I20260925 01:30:14.910823 4665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:30:14.910946 4665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:30:14.911161 4665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:30:14.915375 4665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } }
I20260925 01:30:14.915926 4665 leader_election.cc:302] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6634b23a1e9948f589cd033f3df7380b; no voters:
I20260925 01:30:14.916654 4665 leader_election.cc:288] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:30:14.916810 4669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:30:14.918157 4669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [term 2 LEADER]: Becoming Leader. State: Replica: 6634b23a1e9948f589cd033f3df7380b, State: Running, Role: LEADER
I20260925 01:30:14.918861 4669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } }
I20260925 01:30:14.919241 4665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:30:14.921640 4671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6634b23a1e9948f589cd033f3df7380b. Latest consensus state: current_term: 2 leader_uuid: "6634b23a1e9948f589cd033f3df7380b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } } }
I20260925 01:30:14.921991 4671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:14.921751 4670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6634b23a1e9948f589cd033f3df7380b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6634b23a1e9948f589cd033f3df7380b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44219 } } }
I20260925 01:30:14.922263 4670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:14.927677 4675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:30:14.938781 4675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:30:14.940843 4675 catalog_manager.cc:1259] Loaded cluster ID: f1209f9319384d4883e35dfd0256a108
I20260925 01:30:14.941016 4675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:30:14.946822 4675 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:30:14.948976 4675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6634b23a1e9948f589cd033f3df7380b: Loaded TSK: 0
I20260925 01:30:14.951485 4675 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:30:15.223908 4667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:15.224257 4667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:15.224424 4667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:15.231846 4667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:15.232015 4667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:15.232242 4667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:30:15.242728 4667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.1.keytab
--principal=oryx/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:42227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=38607
--tserver_master_addrs=127.27.160.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:15.244266 4667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:15.245702 4667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:15.260854 4691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:15.261408 4694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:15.261444 4692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:15.262902 4667 server_base.cc:1034] running on GCE node
I20260925 01:30:15.263689 4667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:15.264961 4667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:15.266232 4667 hybrid_clock.cc:648] HybridClock initialized: now 1790299815266158 us; error 61 us; skew 500 ppm
I20260925 01:30:15.266644 4667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:15 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299815, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:15.271035 4667 init.cc:377] Logged in from keytab as oryx/127.27.160.1@KRBTEST.COM (short username oryx)
I20260925 01:30:15.273036 4667 webserver.cc:460] Webserver started at http://127.27.160.1:38607/ using document root <none> and password file <none>
I20260925 01:30:15.274063 4667 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:15.274235 4667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:15.282327 4667 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:30:15.286140 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:15.287848 4667 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:30:15.288091 4667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "be5f6c22b33a480f99cd00896778725d"
format_stamp: "Formatted at 2026-09-25 01:30:12 on dist-test-slave-h6hf"
server_key: "2240ac57ee61c22e0dc56c7f402c0e8b"
server_key_iv: "f82918db4b591fd9a3ac24cbfb529992"
server_key_version: "encryptionkey@0"
I20260925 01:30:15.289553 4667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:15.328967 4667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:15.334618 4667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:15.334913 4667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:15.336095 4667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:15.337805 4667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:15.337895 4667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:15.337980 4667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:15.338037 4667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:15.373085 4667 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:42227
I20260925 01:30:15.373168 4813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:42227 every 8 connection(s)
I20260925 01:30:15.374761 4667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:30:15.385319 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 4667
I20260925 01:30:15.386088 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:42939
--local_ip_for_outbound_sockets=127.27.160.2
--tserver_master_addrs=127.27.160.62:44219
--webserver_port=38561
--webserver_interface=127.27.160.2
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.160.2
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:30:15 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:15 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:30:15.726221 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:15.726526 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:15.726689 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:15.735559 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:15.735699 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:15.735889 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:30:15.747061 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.2.keytab
--principal=oryx/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:42939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=38561
--tserver_master_addrs=127.27.160.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:15.748485 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:15.750214 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:15.770249 4825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:15.770249 4824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:15.771175 4827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:15.772996 4818 server_base.cc:1034] running on GCE node
I20260925 01:30:15.774197 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:15.775923 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:15.777264 4818 hybrid_clock.cc:648] HybridClock initialized: now 1790299815777182 us; error 76 us; skew 500 ppm
I20260925 01:30:15.777827 4818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:15 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299815, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:15.782246 4818 init.cc:377] Logged in from keytab as oryx/127.27.160.2@KRBTEST.COM (short username oryx)
I20260925 01:30:15.783892 4818 webserver.cc:460] Webserver started at http://127.27.160.2:38561/ using document root <none> and password file <none>
I20260925 01:30:15.784627 4818 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:15.784776 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:15.791464 4818 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 01:30:15.795476 4834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:15.797588 4818 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:30:15.797811 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "30a2e77002d64dc5840b80e11fe3595b"
format_stamp: "Formatted at 2026-09-25 01:30:13 on dist-test-slave-h6hf"
server_key: "a2bfa025589379f1d4a8ae0fc000e9ff"
server_key_iv: "49b17c553dcd9423c9a7af996e820048"
server_key_version: "encryptionkey@0"
I20260925 01:30:15.798934 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:15.824247 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:15.829238 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:15.829563 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:15.830793 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:15.832443 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:15.832577 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:15.832726 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:15.832824 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:15.883378 4818 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:42939
I20260925 01:30:15.883455 4946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:42939 every 8 connection(s)
I20260925 01:30:15.885042 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:30:15.892239 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 4818
I20260925 01:30:15.893055 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:40303
--local_ip_for_outbound_sockets=127.27.160.3
--tserver_master_addrs=127.27.160.62:44219
--webserver_port=32863
--webserver_interface=127.27.160.3
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.160.3
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:30:15 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:15 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:30:16.250337 4951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:16.250715 4951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:16.250905 4951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:16.258536 4951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:16.258677 4951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:16.258863 4951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:30:16.268766 4951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/krb5kdc/oryx_127.27.160.3.keytab
--principal=oryx/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:40303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=32863
--tserver_master_addrs=127.27.160.62:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:16.270219 4951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:16.271628 4951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:16.289058 4957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:16.289343 4958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:16.306898 4961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:16.307526 4951 server_base.cc:1034] running on GCE node
I20260925 01:30:16.308243 4951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:16.309397 4951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:16.310683 4951 hybrid_clock.cc:648] HybridClock initialized: now 1790299816310618 us; error 60 us; skew 500 ppm
I20260925 01:30:16.311161 4951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:16 dist-test-slave-h6hf krb5kdc[4079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299816, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:16.314976 4951 init.cc:377] Logged in from keytab as oryx/127.27.160.3@KRBTEST.COM (short username oryx)
I20260925 01:30:16.321498 4951 webserver.cc:460] Webserver started at http://127.27.160.3:32863/ using document root <none> and password file <none>
I20260925 01:30:16.322342 4951 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:16.322521 4951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:16.328018 4951 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 01:30:16.330914 4967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:16.332721 4951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:30:16.332932 4951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "39a6f8779e3c4603901c03abe86bf586"
format_stamp: "Formatted at 2026-09-25 01:30:13 on dist-test-slave-h6hf"
server_key: "c6046cefc4b36269987583903b866c0f"
server_key_iv: "3ca0907c076b02d1ec04e0d86362fc37"
server_key_version: "encryptionkey@0"
I20260925 01:30:16.334213 4951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:16.360014 4951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:16.365525 4951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:16.365839 4951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:16.367035 4951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:16.368690 4951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:16.368809 4951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:16.368939 4951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:16.369022 4951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:16.440582 4951 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:40303
I20260925 01:30:16.440711 5079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:40303 every 8 connection(s)
I20260925 01:30:16.443318 4951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:30:16.446817 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 4951
Sep 25 01:30:16 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:16 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:16.551684 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:16.449869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.27.160.62:44219)
0925 01:30:16.499031 (+ 49162us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:16.499041 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:16.510898 (+ 11857us) server_negotiation.cc:407] Connection header received
0925 01:30:16.512350 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:16.512362 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:16.512503 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:16.512673 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:16.520017 (+ 7344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.521883 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.531831 (+ 9948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.532805 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.532952 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:16.538669 (+ 5717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:16.538686 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:16.538723 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:16.538795 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:16.544634 (+ 5839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:16.545661 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:16.545677 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:16.545695 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:16.546057 (+ 362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:16.546646 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:16.546655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:16.546661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:16.547653 (+ 992us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:16.548005 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:16.549374 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0925 01:30:16.550133 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48992,"thread_start_us":49321,"threads_started":1}
W20260925 01:30:16.842773 4816 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.454s user 0.002s sys 0.013s
W20260925 01:30:16.844782 4817 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:15.377896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:58811 (local address 127.27.160.62:44219)
0925 01:30:15.378680 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:15.378691 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:15.378967 (+ 276us) server_negotiation.cc:407] Connection header received
0925 01:30:15.381033 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:15.381051 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:15.381476 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:15.381792 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:15.383159 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:15.384373 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:15.386848 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:15.387568 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:15.387709 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:16.844309 (+1456600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:58811: BlockingRecv error: recv got EOF from 127.27.160.1:58811 (error 108)
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":355,"threads_started":1}
I20260925 01:30:16.845307 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:15.377924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:58811)
0925 01:30:15.378739 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:15.378798 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:15.380510 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:15.381939 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:15.381946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:15.382441 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:15.382972 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:15.382990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:15.384543 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:15.384546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:15.386697 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:15.386717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:15.387817 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:15.387826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:15.388144 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:15.388871 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:15.388906 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:16.843836 (+1454930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":335,"threads_started":1}
W20260925 01:30:16.846809 4814 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:44219 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.160.62:44219: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Sep 25 01:30:16 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:16 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:30:16.867002 4817 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:16.848217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:52763 (local address 127.27.160.62:44219)
0925 01:30:16.848450 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:16.848459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:16.849176 (+ 717us) server_negotiation.cc:407] Connection header received
0925 01:30:16.850389 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:16.850396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:16.850480 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:16.850558 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:16.852688 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.853987 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.857921 (+ 3934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.858651 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.858760 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:16.866642 (+ 7882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:52763: BlockingRecv error: recv got EOF from 127.27.160.1:52763 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:30:16.868019 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:16.847751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:52763)
0925 01:30:16.849031 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:16.849056 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:16.849990 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:16.850857 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:16.850863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:16.851347 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:16.851568 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:16.851580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.854233 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.854239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:16.855875 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:16.855893 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.859132 (+ 3239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.859139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:16.859383 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:16.861299 (+ 1916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:16.861337 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:16.867567 (+ 6230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1118,"thread_start_us":11958,"threads_started":1}
Sep 25 01:30:16 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:16 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:16.895717 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:16.869008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:45953)
0925 01:30:16.869258 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:16.869281 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:16.869412 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:16.870039 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:16.870047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:16.870814 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:16.871239 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:16.871256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.873055 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.873064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:16.875102 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:16.875125 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.876631 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.876640 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:16.876912 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:16.877882 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:16.877915 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:16.895319 (+ 17404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 01:30:16.895882 4817 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:16.869415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:45953 (local address 127.27.160.62:44219)
0925 01:30:16.869622 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:16.869631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:16.869654 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:30:16.869726 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:16.869734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:16.869837 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:16.869980 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:16.871474 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.872840 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.875335 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.876167 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.876332 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:16.895588 (+ 19256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:45953: BlockingRecv error: recv got EOF from 127.27.160.1:45953 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 01:30:16.896745 4814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:30:17.400440 4950 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.497s user 0.005s sys 0.015s
I20260925 01:30:17.402385 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:15.888680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:40145)
0925 01:30:15.889619 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:15.889678 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:15.891612 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:15.892792 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:15.892805 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:15.893729 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:15.894865 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:15.894988 (+ 123us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:15.897758 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:15.897766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:15.900228 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:15.900250 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:15.901879 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:15.901901 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:15.902277 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:15.903200 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:15.903240 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:17.401385 (+1498145us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":415,"threads_started":1}
W20260925 01:30:17.403293 4949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:15.888096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:40145 (local address 127.27.160.62:44219)
0925 01:30:15.888840 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:15.888853 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:15.889841 (+ 988us) server_negotiation.cc:407] Connection header received
0925 01:30:15.892086 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:15.892096 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:15.892266 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:15.892446 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:15.895449 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:15.897513 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:15.900430 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:15.901561 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:15.901849 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.403091 (+1501242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:40145: BlockingRecv error: recv got EOF from 127.27.160.2:40145 (error 108)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":293,"threads_started":1}
W20260925 01:30:17.403780 4947 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:44219 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.160.62:44219: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:17.420261 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:17.404413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:33183)
0925 01:30:17.404717 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:17.404745 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:17.404934 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:17.405355 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:17.405364 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:17.406065 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:17.406380 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.406397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.407461 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.407470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.409856 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.409879 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.410615 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.410622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.410785 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.415541 (+ 4756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:17.415561 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:17.419934 (+ 4373us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 01:30:17.420446 4949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:17.404544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:33183 (local address 127.27.160.62:44219)
0925 01:30:17.404747 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:17.404751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:17.404839 (+ 88us) server_negotiation.cc:407] Connection header received
0925 01:30:17.405082 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:17.405085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:17.405192 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:17.405270 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:17.406567 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.407301 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.410039 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.410427 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.410565 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.420215 (+ 9650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:33183: BlockingRecv error: recv got EOF from 127.27.160.2:33183 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:17.432830 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:17.421367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:43909)
0925 01:30:17.421540 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:17.421565 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:17.421695 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:17.422410 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:17.422414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:17.422723 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:17.422884 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.422893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.424957 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.424963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.426344 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.426382 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.427675 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.427679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.427803 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.428573 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:17.428592 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:17.432628 (+ 4036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:30:17.433296 4947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:30:17.433614 4949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:17.421714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:43909 (local address 127.27.160.62:44219)
0925 01:30:17.421962 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:17.421972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:17.421998 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:30:17.422085 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:17.422093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:17.422225 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:17.422382 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:17.423036 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.424743 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.426540 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.427494 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.427690 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.433031 (+ 5341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:43909: BlockingRecv error: recv got EOF from 127.27.160.2:43909 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 01:30:17.865440 5083 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.328s user 0.012s sys 0.012s
W20260925 01:30:17.866768 5084 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:16.449407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:59629 (local address 127.27.160.62:44219)
0925 01:30:16.498496 (+ 49089us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:16.498510 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:16.502236 (+ 3726us) server_negotiation.cc:407] Connection header received
0925 01:30:16.516439 (+ 14203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:16.516450 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:16.516578 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:16.516690 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:16.526182 (+ 9492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.527956 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.534464 (+ 6508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.535141 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.535239 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.866453 (+1331214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:59629: BlockingRecv error: recv got EOF from 127.27.160.3:59629 (error 108)
Metrics: {"server-negotiator.queue_time_us":48813,"thread_start_us":49593,"threads_started":1}
I20260925 01:30:17.867352 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:16.448395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:59629)
0925 01:30:16.500323 (+ 51928us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:16.500450 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:16.509376 (+ 8926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:16.517868 (+ 8492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:16.517889 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:16.519167 (+ 1278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:16.525813 (+ 6646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:16.525862 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.528315 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.528325 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:16.531530 (+ 3205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:16.531556 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:16.535674 (+ 4118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:16.535683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:16.536068 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:16.536926 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:16.536970 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:17.866238 (+1329268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51288,"thread_start_us":54702,"threads_started":1}
W20260925 01:30:17.868229 5080 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:44219 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.160.62:44219: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:17.878003 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:17.868934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:46015)
0925 01:30:17.869161 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:17.869183 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:17.869308 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:17.869647 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:17.869651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:17.870017 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:17.870186 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.870195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.871261 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.871269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.872230 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.872241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.872945 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.872951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.873180 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.873785 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:17.873802 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:17.877694 (+ 3892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 01:30:17.878223 5084 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:17.869044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:46015 (local address 127.27.160.62:44219)
0925 01:30:17.869245 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:17.869251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:17.869268 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:30:17.869431 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:17.869440 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:17.869497 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:17.869596 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:17.870325 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.871120 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.872368 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.872773 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.872917 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.877897 (+ 4980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:46015: BlockingRecv error: recv got EOF from 127.27.160.3:46015 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:17.887776 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:17.879005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:37261)
0925 01:30:17.879145 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:17.879164 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:17.879255 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:17.879641 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:17.879649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:17.880030 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:17.880200 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.880209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.881290 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.881294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.882280 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.882292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.883307 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.883313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.883477 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.884160 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:17.884190 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:17.887476 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 01:30:17.887928 5084 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:17.879066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:37261 (local address 127.27.160.62:44219)
0925 01:30:17.879330 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:17.879334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:17.879350 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:30:17.879393 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:17.879397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:17.879454 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:17.879540 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:17.880356 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.881099 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.882423 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.883138 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.883272 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.887666 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:37261: BlockingRecv error: recv got EOF from 127.27.160.3:37261 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
W20260925 01:30:17.888240 5080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:17 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:17.906108 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:17.897233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:38459)
0925 01:30:17.897743 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:17.897759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:17.897870 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:17.898182 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:17.898185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:17.898565 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:17.898723 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.898731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.899683 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.899686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.900612 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:17.900624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.901326 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.901333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:17.901488 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.902213 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:17.902245 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:17.905891 (+ 3646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":288,"threads_started":1}
W20260925 01:30:17.906312 5084 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:17.897337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:38459 (local address 127.27.160.62:44219)
0925 01:30:17.897545 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:17.897549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:17.897835 (+ 286us) server_negotiation.cc:407] Connection header received
0925 01:30:17.897968 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:17.897972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:17.898058 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:17.898142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:17.898850 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.899559 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.900734 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:17.901159 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:17.901281 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:17.906114 (+ 4833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:38459: BlockingRecv error: recv got EOF from 127.27.160.1:38459 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 01:30:18 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:18 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:18.443359 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:18.433877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:43263)
0925 01:30:18.434511 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:18.434550 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:18.434664 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:18.434966 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:18.434969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:18.435357 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:18.435505 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:18.435513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.436516 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.436520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:18.437527 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:18.437578 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.438357 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.438362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:18.438483 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:18.439124 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:18.439140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:18.443135 (+ 3995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":435,"threads_started":1}
W20260925 01:30:18.443588 5089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:18.433916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:43263 (local address 127.27.160.62:44219)
0925 01:30:18.434479 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:18.434486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:18.434629 (+ 143us) server_negotiation.cc:407] Connection header received
0925 01:30:18.434766 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:18.434770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:18.434833 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:18.434922 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:18.435649 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.436389 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.437766 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.438198 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.438330 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:18.443336 (+ 5006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:43263: BlockingRecv error: recv got EOF from 127.27.160.2:43263 (error 108)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":308,"threads_started":1}
Sep 25 01:30:18 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:18 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:18.898454 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:18.888937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:37649)
0925 01:30:18.889796 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:18.889813 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:18.889933 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:18.890229 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:18.890232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:18.890719 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:18.890917 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:18.890927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.892113 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.892119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:18.893247 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:18.893259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.894016 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.894020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:18.894156 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:18.894751 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:18.894769 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:18.898219 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":612,"threads_started":1}
W20260925 01:30:18.898686 5089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:18.889016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:37649 (local address 127.27.160.62:44219)
0925 01:30:18.889359 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:18.889369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:18.889907 (+ 538us) server_negotiation.cc:407] Connection header received
0925 01:30:18.890057 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:18.890061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:18.890126 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:18.890215 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:18.891069 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.891920 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.893402 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.893852 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.893959 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:18.898447 (+ 4488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:37649: BlockingRecv error: recv got EOF from 127.27.160.3:37649 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Sep 25 01:30:18 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:18 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:18.915930 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:18.907036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:38271)
0925 01:30:18.907541 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:18.907557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:18.907669 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:18.907941 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:18.907944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:18.908333 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:18.908484 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:18.908492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.909750 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.909754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:18.910684 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:18.910695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.911377 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.911381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:18.911499 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:18.912147 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:18.912164 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:18.915600 (+ 3436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":281,"threads_started":1}
W20260925 01:30:18.916143 5089 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:18.907085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:38271 (local address 127.27.160.62:44219)
0925 01:30:18.907246 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:18.907250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:18.907626 (+ 376us) server_negotiation.cc:407] Connection header received
0925 01:30:18.907764 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:18.907767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:18.907825 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:18.907902 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:18.908656 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.909618 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.910820 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:18.911251 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:18.911350 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:18.915827 (+ 4477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:38271: BlockingRecv error: recv got EOF from 127.27.160.1:38271 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 01:30:19 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:19 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:19.453765 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:19.444446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:33469)
0925 01:30:19.444910 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:19.444930 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:19.445031 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:19.445535 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:19.445538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:19.445926 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:19.446102 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:19.446110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.447360 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.447364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:19.448264 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:19.448275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.449037 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.449042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:19.449189 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:19.449769 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:19.449787 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:19.453475 (+ 3688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":256,"threads_started":1}
W20260925 01:30:19.454070 5094 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:19.444513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:33469 (local address 127.27.160.62:44219)
0925 01:30:19.445250 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:19.445254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:19.445268 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:30:19.445310 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:19.445313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:19.445380 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:19.445466 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:19.446269 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.447231 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.448414 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.448878 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.448983 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:19.453748 (+ 4765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:33469: BlockingRecv error: recv got EOF from 127.27.160.2:33469 (error 108)
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":468,"threads_started":1}
Sep 25 01:30:19 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:19 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:19.909072 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:19.899388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:47885)
0925 01:30:19.899929 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:19.899944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:19.900110 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:19.900454 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:19.900460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:19.900912 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:19.901093 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:19.901102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.902313 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.902318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:19.903336 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:19.903347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.904159 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.904165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:19.904349 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:19.905030 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:19.905055 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:19.908831 (+ 3776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":279,"threads_started":1}
W20260925 01:30:19.909312 5094 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:19.899611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:47885 (local address 127.27.160.62:44219)
0925 01:30:19.899828 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:19.899834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:19.900023 (+ 189us) server_negotiation.cc:407] Connection header received
0925 01:30:19.900243 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:19.900254 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:19.900317 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:19.900418 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:19.901285 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.902168 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.903467 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.903988 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.904143 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:19.909013 (+ 4870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:47885: BlockingRecv error: recv got EOF from 127.27.160.3:47885 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 01:30:19 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:19 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:19.926728 5096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:19.917227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:34437)
0925 01:30:19.917802 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:19.917818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:19.917932 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:19.918302 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:19.918336 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:19.918825 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:19.919024 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:19.919035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.920186 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.920192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:19.921246 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:19.921260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.922011 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.922017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:19.922227 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:19.922832 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:19.922859 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:19.926366 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":316,"threads_started":1}
W20260925 01:30:19.926800 5094 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:19.917243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:34437 (local address 127.27.160.62:44219)
0925 01:30:19.917489 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:19.917500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:19.917908 (+ 408us) server_negotiation.cc:407] Connection header received
0925 01:30:19.918032 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:19.918040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:19.918140 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:19.918268 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:19.919178 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.920026 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.921404 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:19.921852 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:19.921999 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:19.926560 (+ 4561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:34437: BlockingRecv error: recv got EOF from 127.27.160.1:34437 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 01:30:20 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:20 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:20.463657 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:20.454729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:53931)
0925 01:30:20.455250 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:20.455277 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:20.455418 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:20.455754 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:20.455758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:20.456143 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:20.456305 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:20.456314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.457514 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.457518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:20.458413 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:20.458425 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.459123 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.459127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:20.459261 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:20.459861 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:20.459878 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:20.463400 (+ 3522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":285,"threads_started":1}
W20260925 01:30:20.463999 5098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:20.454726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:53931 (local address 127.27.160.62:44219)
0925 01:30:20.455261 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:20.455267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:20.455351 (+ 84us) server_negotiation.cc:407] Connection header received
0925 01:30:20.455540 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:20.455544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:20.455614 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:20.455702 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:20.456465 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.457351 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.458554 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.458983 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.459080 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:20.463660 (+ 4580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:53931: BlockingRecv error: recv got EOF from 127.27.160.2:53931 (error 108)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":293,"threads_started":1}
Sep 25 01:30:20 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:20 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:20.919683 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:20.910028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:54119)
0925 01:30:20.910670 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:20.910688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:20.910854 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:20.911263 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:20.911268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:20.911700 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:20.911893 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:20.911903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.913010 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.913017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:20.914049 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:20.914062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.914816 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.914823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:20.915031 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:20.915620 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:20.915639 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:20.919340 (+ 3701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":392,"threads_started":1}
W20260925 01:30:20.919993 5098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:20.910332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:54119 (local address 127.27.160.62:44219)
0925 01:30:20.910553 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:20.910562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:20.910773 (+ 211us) server_negotiation.cc:407] Connection header received
0925 01:30:20.911014 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:20.911022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:20.911115 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:20.911225 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:20.912054 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.912849 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.914215 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.914657 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.914804 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:20.919582 (+ 4778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:54119: BlockingRecv error: recv got EOF from 127.27.160.3:54119 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 01:30:20 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:20 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:20.936923 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:20.927695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:33519)
0925 01:30:20.928210 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:20.928225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:20.928375 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:20.928734 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:20.928739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:20.929196 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:20.929359 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:20.929368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.930371 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.930378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:20.931662 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:20.931695 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.932446 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.932450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:20.932591 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:20.933257 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:20.933274 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:20.936684 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":296,"threads_started":1}
W20260925 01:30:20.937223 5098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:20.927770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:33519 (local address 127.27.160.62:44219)
0925 01:30:20.928034 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:20.928041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:20.928306 (+ 265us) server_negotiation.cc:407] Connection header received
0925 01:30:20.928493 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:20.928498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:20.928596 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:20.928700 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:20.929499 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.930228 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.931820 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:20.932287 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:20.932436 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:20.936847 (+ 4411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:33519: BlockingRecv error: recv got EOF from 127.27.160.1:33519 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 25 01:30:21 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:21 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:21.473842 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:21.464720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:44145)
0925 01:30:21.465251 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:21.465273 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:21.465386 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:21.465761 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:21.465765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:21.466140 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:21.466305 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:21.466314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.467406 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.467414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:21.468593 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:21.468606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.469434 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.469438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:21.469581 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:21.470205 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:21.470221 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:21.473587 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":301,"threads_started":1}
W20260925 01:30:21.474012 5102 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:21.464802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:44145 (local address 127.27.160.62:44219)
0925 01:30:21.465285 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:21.465289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:21.465341 (+ 52us) server_negotiation.cc:407] Connection header received
0925 01:30:21.465540 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:21.465544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:21.465606 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:21.465711 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:21.466441 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.467276 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.468738 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.469307 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.469400 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:21.473788 (+ 4388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:44145: BlockingRecv error: recv got EOF from 127.27.160.2:44145 (error 108)
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
Sep 25 01:30:21 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:21 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:21.929715 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:21.920732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:47179)
0925 01:30:21.921370 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:21.921393 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:21.921536 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:21.921927 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:21.921933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:21.922497 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:21.922677 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:21.922686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.923709 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.923712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:21.924671 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:21.924681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.925328 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.925332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:21.925448 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:21.926038 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:21.926055 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:21.929447 (+ 3392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":379,"threads_started":1}
W20260925 01:30:21.929977 5102 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:21.920828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:47179 (local address 127.27.160.62:44219)
0925 01:30:21.921044 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:21.921050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:21.921485 (+ 435us) server_negotiation.cc:407] Connection header received
0925 01:30:21.921681 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:21.921685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:21.921768 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:21.921880 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:21.922836 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.923588 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.924796 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.925214 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.925304 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:21.929698 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:47179: BlockingRecv error: recv got EOF from 127.27.160.3:47179 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 01:30:21 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:21 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:21.946709 5104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:21.937769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:43731)
0925 01:30:21.938268 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:21.938285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:21.938393 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:21.938718 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:21.938722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:21.939102 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:21.939253 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:21.939261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.940362 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.940366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:21.941347 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:21.941359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.942022 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.942027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:21.942155 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:21.942703 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:21.942719 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:21.946367 (+ 3648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
W20260925 01:30:21.946844 5102 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:21.937943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:43731 (local address 127.27.160.62:44219)
0925 01:30:21.938189 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:21.938198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:21.938368 (+ 170us) server_negotiation.cc:407] Connection header received
0925 01:30:21.938492 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:21.938496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:21.938601 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:21.938690 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:21.939384 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.940235 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.941471 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:21.941887 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:21.941996 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:21.946587 (+ 4591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:43731: BlockingRecv error: recv got EOF from 127.27.160.1:43731 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 01:30:22 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:22 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:22.488433 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:22.474743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:38625)
0925 01:30:22.475288 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:22.475308 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:22.475411 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:22.475704 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:22.475707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:22.476142 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:22.476292 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:22.476300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.477285 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.477289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:22.478230 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:22.478242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.479049 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.479058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:22.479219 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:22.484660 (+ 5441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:22.484680 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:22.488199 (+ 3519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":324,"threads_started":1}
W20260925 01:30:22.488838 5106 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:22.474922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:38625 (local address 127.27.160.62:44219)
0925 01:30:22.475398 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:22.475402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:22.475418 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:30:22.475517 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:22.475520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:22.475583 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:22.475663 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:22.476417 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.477176 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.478350 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.478882 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.478995 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:22.488411 (+ 9416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:38625: BlockingRecv error: recv got EOF from 127.27.160.2:38625 (error 108)
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":285,"threads_started":1}
Sep 25 01:30:22 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:22 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:22.940629 5107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:22.930802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:60547)
0925 01:30:22.931319 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:22.931334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:22.931500 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:22.931854 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:22.931858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:22.932334 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:22.932482 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:22.932491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.933714 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.933722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:22.934792 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:22.934804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.935498 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.935504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:22.935707 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:22.936361 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:22.936380 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:22.940289 (+ 3909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
W20260925 01:30:22.940759 5106 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:22.930894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:60547 (local address 127.27.160.62:44219)
0925 01:30:22.931131 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:22.931139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:22.931428 (+ 289us) server_negotiation.cc:407] Connection header received
0925 01:30:22.931650 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:22.931654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:22.931716 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:22.931821 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:22.932632 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.933584 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.934934 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.935352 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.935486 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:22.940514 (+ 5028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:60547: BlockingRecv error: recv got EOF from 127.27.160.3:60547 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 01:30:22 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:22 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:22.956341 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:22.947797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:60363)
0925 01:30:22.948332 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:22.948348 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:22.948459 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:22.948753 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:22.948756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:22.949229 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:22.949393 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:22.949401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.950377 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.950381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:22.951299 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:22.951311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.951974 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.951978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:22.952122 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:22.952769 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:22.952786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:22.956107 (+ 3321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":311,"threads_started":1}
W20260925 01:30:22.956655 5106 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:22.947995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:60363 (local address 127.27.160.62:44219)
0925 01:30:22.948181 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:22.948186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:22.948418 (+ 232us) server_negotiation.cc:407] Connection header received
0925 01:30:22.948567 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:22.948571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:22.948628 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:22.948712 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:22.949513 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.950257 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.951415 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:22.951848 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:22.951939 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:22.956361 (+ 4422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:60363: BlockingRecv error: recv got EOF from 127.27.160.1:60363 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 01:30:23 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:23 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:23.498737 5109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:23.489497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:59027)
0925 01:30:23.490005 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:23.490026 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:23.490131 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:23.490509 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:23.490512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:23.490891 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:23.491043 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:23.491051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.492091 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.492096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:23.493014 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:23.493025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.494016 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.494022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:23.494213 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:23.494909 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:23.494926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:23.498493 (+ 3567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":278,"threads_started":1}
W20260925 01:30:23.499163 5110 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:23.489583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:59027 (local address 127.27.160.62:44219)
0925 01:30:23.490206 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:23.490211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:23.490229 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:30:23.490281 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:23.490285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:23.490372 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:23.490476 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:23.491204 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.491924 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.493181 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.493847 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.493988 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:23.498837 (+ 4849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:59027: BlockingRecv error: recv got EOF from 127.27.160.2:59027 (error 108)
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":373,"threads_started":1}
Sep 25 01:30:23 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:23 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:23.951530 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:23.941663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:48507)
0925 01:30:23.942206 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:23.942223 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:23.942337 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:23.942699 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:23.942704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:23.943110 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:23.943281 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:23.943289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.944359 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.944365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:23.945581 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:23.945597 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.946347 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.946354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:23.946499 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:23.947169 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:23.947188 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:23.951154 (+ 3966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":315,"threads_started":1}
W20260925 01:30:23.951669 5110 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:23.941821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:48507 (local address 127.27.160.62:44219)
0925 01:30:23.942018 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:23.942023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:23.942299 (+ 276us) server_negotiation.cc:407] Connection header received
0925 01:30:23.942477 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:23.942480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:23.942547 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:23.942642 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:23.943421 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.944153 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.945752 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.946174 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.946274 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:23.951397 (+ 5123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:48507: BlockingRecv error: recv got EOF from 127.27.160.3:48507 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 01:30:23 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:23 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:23.967028 5112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:23.957513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:41597)
0925 01:30:23.958118 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:23.958135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:23.958296 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:23.958683 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:23.958689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:23.959144 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:23.959323 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:23.959332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.960380 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.960393 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:23.961408 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:23.961421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.962166 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.962172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:23.962334 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:23.962979 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:23.962996 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:23.966805 (+ 3809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":263,"threads_started":1}
W20260925 01:30:23.967242 5110 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:23.957613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:41597 (local address 127.27.160.62:44219)
0925 01:30:23.957823 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:23.957832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:23.958224 (+ 392us) server_negotiation.cc:407] Connection header received
0925 01:30:23.958431 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:23.958437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:23.958511 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:23.958639 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:23.959474 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.960225 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.961564 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:23.962010 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:23.962148 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:23.966996 (+ 4848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:41597: BlockingRecv error: recv got EOF from 127.27.160.1:41597 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 01:30:24 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:24 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:24.508867 5113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:24.499594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:56085)
0925 01:30:24.500182 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:24.500202 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:24.500313 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:24.500611 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:24.500614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:24.501039 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:24.501236 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:24.501245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.502277 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.502281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:24.503392 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:24.503405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.504199 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.504204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:24.504320 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:24.504945 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:24.504962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:24.508479 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":278,"threads_started":1}
W20260925 01:30:24.508961 5114 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:24.499671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:56085 (local address 127.27.160.62:44219)
0925 01:30:24.500170 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:24.500174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:24.500273 (+ 99us) server_negotiation.cc:407] Connection header received
0925 01:30:24.500408 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:24.500411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:24.500481 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:24.500575 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:24.501358 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.502130 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.503526 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.504000 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.504169 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:24.508693 (+ 4524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:56085: BlockingRecv error: recv got EOF from 127.27.160.2:56085 (error 108)
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":280,"threads_started":1}
Sep 25 01:30:24 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:24 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:24.962141 5115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:24.952922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:36929)
0925 01:30:24.953598 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:24.953615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:24.953781 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:24.954200 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:24.954207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:24.954814 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:24.955091 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:24.955103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.956220 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.956223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:24.957231 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:24.957242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.957890 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.957894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:24.958031 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:24.958599 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:24.958617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:24.961814 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":302,"threads_started":1}
W20260925 01:30:24.962235 5114 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:24.952994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:36929 (local address 127.27.160.62:44219)
0925 01:30:24.953284 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:24.953294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:24.953728 (+ 434us) server_negotiation.cc:407] Connection header received
0925 01:30:24.953905 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:24.953909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:24.953999 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:24.954130 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:24.955277 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.956097 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.957367 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.957762 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.957857 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:24.962004 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:36929: BlockingRecv error: recv got EOF from 127.27.160.3:36929 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 01:30:24 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:24 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:24.976907 5116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:24.967919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:54933)
0925 01:30:24.968467 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:24.968482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:24.968645 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:24.968977 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:24.968982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:24.969454 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:24.969633 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:24.969641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.970645 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.970651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:24.971681 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:24.971693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.972411 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.972417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:24.972588 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:24.973255 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:24.973273 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:24.976568 (+ 3295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
W20260925 01:30:24.977052 5114 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:24.968136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:54933 (local address 127.27.160.62:44219)
0925 01:30:24.968345 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:24.968351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:24.968577 (+ 226us) server_negotiation.cc:407] Connection header received
0925 01:30:24.968785 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:24.968789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:24.968845 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:24.968947 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:24.969777 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.970495 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.971827 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:24.972229 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:24.972354 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:24.976790 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:54933: BlockingRecv error: recv got EOF from 127.27.160.1:54933 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 01:30:25 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:25 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:25.519277 5117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:25.509846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:44893)
0925 01:30:25.510424 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:25.510445 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:25.510570 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:25.510894 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:25.510897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:25.511265 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:25.511427 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:25.511435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.512607 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.512612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:25.513580 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:25.513592 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.514587 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.514601 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:25.514722 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:25.515510 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:25.515529 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:25.519024 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":272,"threads_started":1}
W20260925 01:30:25.519740 5118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:25.509957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:44893 (local address 127.27.160.62:44219)
0925 01:30:25.510501 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:25.510505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:25.510522 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:30:25.510683 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:25.510687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:25.510767 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:25.510854 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:25.511554 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.512444 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.513734 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.514388 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.514489 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:25.519435 (+ 4946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:44893: BlockingRecv error: recv got EOF from 127.27.160.2:44893 (error 108)
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":288,"threads_started":1}
Sep 25 01:30:25 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:25 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:25.972007 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:25.963169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:57203)
0925 01:30:25.963701 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:25.963718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:25.963835 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:25.964125 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:25.964128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:25.964508 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:25.964693 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:25.964701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.966020 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.966026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:25.967100 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:25.967112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.967756 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.967760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:25.967867 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:25.968430 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:25.968447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:25.971771 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":270,"threads_started":1}
W20260925 01:30:25.972169 5118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:25.963211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:57203 (local address 127.27.160.62:44219)
0925 01:30:25.963387 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:25.963392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:25.963791 (+ 399us) server_negotiation.cc:407] Connection header received
0925 01:30:25.963933 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:25.963937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:25.963999 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:25.964085 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:25.964847 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.965884 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.967228 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.967607 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.967702 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:25.971951 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:57203: BlockingRecv error: recv got EOF from 127.27.160.3:57203 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 01:30:25 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:25 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:25.988946 5120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:25.978092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:56849)
0925 01:30:25.978691 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:25.978720 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:25.979012 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:25.979447 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:25.979453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:25.979996 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:25.980165 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:25.980175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.981509 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.981516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:25.982629 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:25.982643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.983497 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.983504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:25.983734 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:25.984461 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:25.984482 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:25.988507 (+ 4025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":283,"threads_started":1}
W20260925 01:30:25.989068 5118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:25.978144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:56849 (local address 127.27.160.62:44219)
0925 01:30:25.978365 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:25.978373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:25.978910 (+ 537us) server_negotiation.cc:407] Connection header received
0925 01:30:25.979174 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:25.979180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:25.979266 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:25.979418 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:25.980339 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.981355 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.982798 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:25.983313 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:25.983479 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:25.988742 (+ 5263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:56849: BlockingRecv error: recv got EOF from 127.27.160.1:56849 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 01:30:26 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:26 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:26.529870 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:26.520247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:60837)
0925 01:30:26.520792 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:26.520814 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:26.520920 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:26.521399 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:26.521402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:26.521802 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:26.521949 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:26.521957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.522959 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.522963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:26.523902 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:26.523913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.524597 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.524602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:26.524722 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:26.525490 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:26.525509 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:26.529627 (+ 4118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":231,"threads_started":1}
W20260925 01:30:26.530059 5122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:26.520263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:60837 (local address 127.27.160.62:44219)
0925 01:30:26.521058 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:26.521062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:26.521078 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:30:26.521171 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:26.521176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:26.521257 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:26.521345 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:26.522087 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.522834 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.524026 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.524459 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.524558 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:26.529818 (+ 5260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:60837: BlockingRecv error: recv got EOF from 127.27.160.2:60837 (error 108)
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":581,"threads_started":1}
Sep 25 01:30:26 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:26 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:26.982093 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:26.972889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:33193)
0925 01:30:26.973380 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:26.973397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:26.973509 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:26.973796 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:26.973800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:26.974217 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:26.974368 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:26.974375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.975413 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.975417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:26.976405 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:26.976417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.977058 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.977062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:26.977209 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:26.977847 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:26.977865 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:26.981867 (+ 4002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
W20260925 01:30:26.982316 5122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:26.973037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:33193 (local address 127.27.160.62:44219)
0925 01:30:26.973282 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:26.973288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:26.973472 (+ 184us) server_negotiation.cc:407] Connection header received
0925 01:30:26.973613 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:26.973617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:26.973675 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:26.973754 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:26.974491 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.975229 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.976533 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.976910 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.977002 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:26.982055 (+ 5053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:33193: BlockingRecv error: recv got EOF from 127.27.160.3:33193 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 25 01:30:26 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:26 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:26.999240 5124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:26.990061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:34669)
0925 01:30:26.990509 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:26.990541 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:26.990655 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:26.990961 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:26.990965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:26.991337 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:26.991487 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:26.991495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.992807 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.992811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:26.994105 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:26.994118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.994826 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.994830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:26.994944 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:26.995512 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:26.995535 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:26.998980 (+ 3445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
W20260925 01:30:26.999439 5122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:26.990163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:34669 (local address 127.27.160.62:44219)
0925 01:30:26.990395 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:26.990401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:26.990619 (+ 218us) server_negotiation.cc:407] Connection header received
0925 01:30:26.990763 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:26.990767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:26.990828 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:26.990908 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:26.991621 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.992655 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.994227 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:26.994700 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:26.994825 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:26.999172 (+ 4347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:34669: BlockingRecv error: recv got EOF from 127.27.160.1:34669 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 25 01:30:27 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:27 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:27.546783 5125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:27.530784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:49923)
0925 01:30:27.531385 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:27.531406 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:27.531529 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:27.531971 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:27.531975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:27.532596 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:27.532850 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:27.532864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:27.534550 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:27.534557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:27.536099 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:27.536116 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:27.537563 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:27.537572 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:27.537778 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:27.543033 (+ 5255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:27.543055 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:27.546541 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":329,"threads_started":1}
W20260925 01:30:27.547065 5126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:27.530924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:49923 (local address 127.27.160.62:44219)
0925 01:30:27.531445 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:27.531449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:27.531466 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:30:27.531664 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:27.531671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:27.531785 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:27.531912 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:27.533062 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:27.534394 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:27.536475 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:27.537180 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:27.537315 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:27.546795 (+ 9480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:49923: BlockingRecv error: recv got EOF from 127.27.160.2:49923 (error 108)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":301,"threads_started":1}
Sep 25 01:30:27 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:27 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:27.992447 5127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:27.983143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:44201)
0925 01:30:27.983653 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:27.983669 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:27.983780 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:27.984101 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:27.984104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:27.984467 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:27.984607 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:27.984615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:27.985835 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:27.985841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:27.986844 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:27.986857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:27.987667 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:27.987671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:27.987778 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:27.988415 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:27.988433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:27.992108 (+ 3675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":285,"threads_started":1}
W20260925 01:30:27.992539 5126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:27.983370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:44201 (local address 127.27.160.62:44219)
0925 01:30:27.983613 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:27.983617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:27.983742 (+ 125us) server_negotiation.cc:407] Connection header received
0925 01:30:27.983893 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:27.983897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:27.983961 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:27.984060 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:27.984745 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:27.985648 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:27.987007 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:27.987498 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:27.987639 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:27.992308 (+ 4669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:44201: BlockingRecv error: recv got EOF from 127.27.160.3:44201 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 25 01:30:28 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:28 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:28.010072 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:28.000363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:57123)
0925 01:30:28.000856 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:28.000872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:28.001047 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:28.001419 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:28.001423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:28.001885 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:28.002079 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:28.002088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.003404 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.003412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:28.004531 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:28.004545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.005303 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.005309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:28.005493 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:28.006138 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:28.006156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:28.009688 (+ 3532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":264,"threads_started":1}
W20260925 01:30:28.010140 5126 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:28.000417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:57123 (local address 127.27.160.62:44219)
0925 01:30:28.000667 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:28.000676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:28.000958 (+ 282us) server_negotiation.cc:407] Connection header received
0925 01:30:28.001195 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:28.001201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:28.001286 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:28.001393 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:28.002222 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.003227 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.004673 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.005091 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.005292 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:28.009885 (+ 4593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:57123: BlockingRecv error: recv got EOF from 127.27.160.1:57123 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 01:30:28 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:28 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:28.556905 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:28.547746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:51045)
0925 01:30:28.548293 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:28.548313 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:28.548415 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:28.548737 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:28.548740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:28.549186 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:28.549359 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:28.549368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.550334 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.550338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:28.551241 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:28.551252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.551904 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.551908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:28.552052 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:28.552667 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:28.552684 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:28.556581 (+ 3897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":313,"threads_started":1}
W20260925 01:30:28.557039 5130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:28.547872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:51045 (local address 127.27.160.62:44219)
0925 01:30:28.548345 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:28.548350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:28.548365 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:30:28.548533 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:28.548536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:28.548596 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:28.548686 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:28.549479 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.550213 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.551367 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.551774 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.551866 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:28.556763 (+ 4897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:51045: BlockingRecv error: recv got EOF from 127.27.160.2:51045 (error 108)
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":281,"threads_started":1}
Sep 25 01:30:28 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:29 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:29.002729 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:28.993495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:50563)
0925 01:30:28.994077 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:28.994094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:28.994208 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:28.994536 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:28.994539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:28.995051 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:28.995275 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:28.995297 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.996455 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.996460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:28.997460 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:28.997472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.998106 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.998111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:28.998232 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:28.998860 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:28.998880 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:29.002477 (+ 3597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":324,"threads_started":1}
W20260925 01:30:29.002985 5130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:28.993538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:50563 (local address 127.27.160.62:44219)
0925 01:30:28.993703 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:28.993708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:28.994197 (+ 489us) server_negotiation.cc:407] Connection header received
0925 01:30:28.994303 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:28.994310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:28.994386 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:28.994492 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:28.995419 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.996263 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.997575 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:28.997959 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:28.998053 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:29.002753 (+ 4700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:50563: BlockingRecv error: recv got EOF from 127.27.160.3:50563 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 01:30:29 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:29 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:29.020628 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:29.011283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:36453)
0925 01:30:29.011845 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:29.011871 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:29.012014 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:29.012366 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:29.012370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:29.012798 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:29.013010 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:29.013021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:29.014066 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:29.014070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:29.015046 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:29.015062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:29.015912 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:29.015916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:29.016071 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:29.016833 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:29.016851 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:29.020345 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":270,"threads_started":1}
W20260925 01:30:29.020776 5130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:29.011350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:36453 (local address 127.27.160.62:44219)
0925 01:30:29.011587 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:29.011593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:29.011957 (+ 364us) server_negotiation.cc:407] Connection header received
0925 01:30:29.012124 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:29.012128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:29.012204 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:29.012287 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:29.013204 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:29.013924 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:29.015183 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:29.015758 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:29.015879 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:29.020555 (+ 4676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:36453: BlockingRecv error: recv got EOF from 127.27.160.1:36453 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 01:30:29 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:29 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:29.567811 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:29.557886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:54753)
0925 01:30:29.558422 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:29.558443 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:29.558557 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:29.558939 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:29.558944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:29.559362 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:29.559565 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:29.559574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:29.560649 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:29.560653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:29.561792 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:29.561805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:29.562580 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:29.562586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:29.562742 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:29.563543 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:29.563562 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:29.567484 (+ 3922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":295,"threads_started":1}
W20260925 01:30:29.568066 5134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:29.557957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:54753 (local address 127.27.160.62:44219)
0925 01:30:29.558417 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:29.558422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:29.558508 (+ 86us) server_negotiation.cc:407] Connection header received
0925 01:30:29.558652 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:29.558656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:29.558727 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:29.558814 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:29.559715 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:29.560492 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:29.561994 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:29.562418 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:29.562517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:29.567803 (+ 5286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:54753: BlockingRecv error: recv got EOF from 127.27.160.2:54753 (error 108)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
Sep 25 01:30:30 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:30 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:30.015230 5135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:30.004156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:35621)
0925 01:30:30.004785 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:30.004807 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:30.004951 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:30.005314 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:30.005318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:30.005749 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:30.005991 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:30.006002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.007231 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.007236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:30.008299 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:30.008314 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.009289 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.009294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:30.009485 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:30.010215 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:30.010243 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:30.014832 (+ 4589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":382,"threads_started":1}
W20260925 01:30:30.015411 5134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:30.004300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:35621 (local address 127.27.160.62:44219)
0925 01:30:30.004507 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:30.004514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:30.004975 (+ 461us) server_negotiation.cc:407] Connection header received
0925 01:30:30.005047 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:30.005051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:30.005154 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:30.005250 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:30.006193 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.007062 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.008519 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.009004 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.009221 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:30.015127 (+ 5906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:35621: BlockingRecv error: recv got EOF from 127.27.160.3:35621 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 01:30:30 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:30 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:30.031855 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:30.021822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:37319)
0925 01:30:30.022383 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:30.022400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:30.022564 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:30.023012 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:30.023017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:30.023454 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:30.023597 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:30.023606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.024667 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.024677 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:30.025668 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:30.025681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.026387 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.026393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:30.026586 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:30.027191 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:30.027208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:30.031608 (+ 4400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":290,"threads_started":1}
W20260925 01:30:30.032122 5134 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:30.021914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:37319 (local address 127.27.160.62:44219)
0925 01:30:30.022170 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:30.022178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:30.022495 (+ 317us) server_negotiation.cc:407] Connection header received
0925 01:30:30.022701 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:30.022705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:30.022780 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:30.022988 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:30.023755 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.024504 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.025808 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.026228 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.026367 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:30.031819 (+ 5452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:37319: BlockingRecv error: recv got EOF from 127.27.160.1:37319 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 01:30:30 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:30 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.2@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:30.579246 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:30.569404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.2:40523)
0925 01:30:30.570243 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:30.570267 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:30.570420 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:30.570810 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:30.570815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:30.571388 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:30.571607 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:30.571618 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.572715 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.572721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:30.573693 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:30.573705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.574451 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.574457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:30.574645 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:30.575315 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:30.575333 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:30.579001 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":304,"threads_started":1}
W20260925 01:30:30.579551 5138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:30.569586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:40523 (local address 127.27.160.62:44219)
0925 01:30:30.570076 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:30.570080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:30.570366 (+ 286us) server_negotiation.cc:407] Connection header received
0925 01:30:30.570557 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:30.570561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:30.570646 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:30.570770 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:30.571757 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.572563 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.573847 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:30.574294 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:30.574427 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:30.579229 (+ 4802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.2:40523: BlockingRecv error: recv got EOF from 127.27.160.2:40523 (error 108)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":282,"threads_started":1}
Sep 25 01:30:31 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:31 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.3@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:31.027266 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:31.016534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.3:42825)
0925 01:30:31.017065 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:31.017087 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:31.017261 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:31.017650 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:31.017654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:31.018092 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:31.018266 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:31.018276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.019703 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.019708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:31.020914 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:31.020928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.021769 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.021774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:31.021946 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:31.022666 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:31.022686 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:31.027023 (+ 4337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":308,"threads_started":1}
W20260925 01:30:31.027527 5138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:31.016682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:42825 (local address 127.27.160.62:44219)
0925 01:30:31.016864 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:31.016869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:31.017253 (+ 384us) server_negotiation.cc:407] Connection header received
0925 01:30:31.017374 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:31.017378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:31.017446 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:31.017545 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:31.018481 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.019518 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.021097 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.021597 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.021729 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:31.027261 (+ 5532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.3:42825: BlockingRecv error: recv got EOF from 127.27.160.3:42825 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 01:30:31 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:30:31 dist-test-slave-h6hf krb5kdc[4079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.160.1@KRBTEST.COM for oryx/127.27.160.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:30:31.042938 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:31.033244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44219 (local address 127.27.160.1:52163)
0925 01:30:31.033871 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:31.033942 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:31.034108 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:31.034508 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:31.034516 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:31.034908 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:31.035075 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:31.035084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.036146 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.036150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:31.037378 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:31.037392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.038290 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.038296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:31.038468 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:31.039121 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:31.039139 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:31.042688 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.160.62:44219: Server oryx/127.27.160.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":376,"threads_started":1}
W20260925 01:30:31.043187 5138 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:31.033350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:52163 (local address 127.27.160.62:44219)
0925 01:30:31.033696 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:31.033704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:31.034035 (+ 331us) server_negotiation.cc:407] Connection header received
0925 01:30:31.034269 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:31.034273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:31.034345 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:31.034472 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:31.035221 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.035981 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.037516 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.038132 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.038270 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:31.042872 (+ 4602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.160.1:52163: BlockingRecv error: recv got EOF from 127.27.160.1:52163 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
I20260925 01:30:31.451028 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4667
I20260925 01:30:31.464252 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:31.452297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52314 (local address 127.27.160.1:42227)
0925 01:30:31.453057 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:31.453065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:31.453097 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:30:31.453237 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:31.453244 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:31.453510 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:31.453702 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:31.455179 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.456355 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.457653 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.458245 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.458357 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:31.461032 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:31.461042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:31.461058 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:31.461104 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:31.463698 (+ 2594us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.160.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:30:31.464016 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52314: Request ticket server kudu/127.27.160.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":418,"threads_started":1}
W20260925 01:30:31.464387 5142 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52314: Request ticket server kudu/127.27.160.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:30:31.464485 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4667: Not authorized: Client connection negotiation failed: client connection to 127.27.160.1:42227: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.160.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:30:31.464736 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 4667
I20260925 01:30:31.482647 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4818
I20260925 01:30:31.495537 5143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:31.483884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.27.160.2:42939)
0925 01:30:31.484505 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:31.484513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:31.484540 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:30:31.484607 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:31.484614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:31.484851 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:31.485075 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:31.486554 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.487578 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.488851 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.489510 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.489679 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:31.492170 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:31.492180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:31.492195 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:31.492233 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:31.494822 (+ 2589us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.160.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:30:31.495188 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44462: Request ticket server kudu/127.27.160.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":314,"threads_started":1}
W20260925 01:30:31.495604 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4818: Not authorized: Client connection negotiation failed: client connection to 127.27.160.2:42939: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.160.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:30:31.495657 5143 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44462: Request ticket server kudu/127.27.160.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:30:31.495774 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 4818
I20260925 01:30:31.512418 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4951
I20260925 01:30:31.526527 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:31.513725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.27.160.3:40303)
0925 01:30:31.514439 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:31.514447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:31.514483 (+ 36us) server_negotiation.cc:407] Connection header received
0925 01:30:31.514556 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:31.514563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:31.514828 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:31.515140 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:31.516631 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.517851 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.519113 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:31.519772 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:31.519934 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:31.522942 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:31.522953 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:31.522969 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:31.523024 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:31.525739 (+ 2715us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.160.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:30:31.526160 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42156: Request ticket server kudu/127.27.160.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":380,"threads_started":1}
W20260925 01:30:31.526578 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4951: Not authorized: Client connection negotiation failed: client connection to 127.27.160.3:40303: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.160.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:30:31.526659 5144 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42156: Request ticket server kudu/127.27.160.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:30:31.526783 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 4951
I20260925 01:30:31.545377 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 4598
I20260925 01:30:31.546911 4660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:31.664239 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 4598
2026-09-25T01:30:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21499 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T01:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:30:31Z Disabled control of system clock
I20260925 01:30:31.741391 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40107
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:40107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 01:30:32.068379 5149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:32.068678 5149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:32.068759 5149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:32.076180 5149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:30:32.076303 5149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:32.076344 5149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:32.076395 5149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:30:32.076431 5149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:30:32.085958 5149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:40107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:32.087327 5149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:32.089090 5149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:32.101832 5154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:32.101941 5157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:32.101898 5155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:32.103045 5149 server_base.cc:1034] running on GCE node
I20260925 01:30:32.103876 5149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:32.105460 5149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:32.106681 5149 hybrid_clock.cc:648] HybridClock initialized: now 1790299832106637 us; error 31 us; skew 500 ppm
I20260925 01:30:32.107071 5149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:30:32.109531 5149 webserver.cc:460] Webserver started at http://127.27.160.62:37447/ using document root <none> and password file <none>
I20260925 01:30:32.110291 5149 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:32.110390 5149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:32.110766 5149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:32.113862 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "c3f279061ad24cf7b32fb65f22400805"
format_stamp: "Formatted at 2026-09-25 01:30:32 on dist-test-slave-h6hf"
server_key: "896e14134e5dd66657b52bc9ab4ae15f"
server_key_iv: "920d9fe5357d7896b5c6214c060f7019"
server_key_version: "encryptionkey@0"
I20260925 01:30:32.114630 5149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "c3f279061ad24cf7b32fb65f22400805"
format_stamp: "Formatted at 2026-09-25 01:30:32 on dist-test-slave-h6hf"
server_key: "896e14134e5dd66657b52bc9ab4ae15f"
server_key_iv: "920d9fe5357d7896b5c6214c060f7019"
server_key_version: "encryptionkey@0"
I20260925 01:30:32.119691 5149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:30:32.123205 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:32.124608 5149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:30:32.124796 5149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "c3f279061ad24cf7b32fb65f22400805"
format_stamp: "Formatted at 2026-09-25 01:30:32 on dist-test-slave-h6hf"
server_key: "896e14134e5dd66657b52bc9ab4ae15f"
server_key_iv: "920d9fe5357d7896b5c6214c060f7019"
server_key_version: "encryptionkey@0"
I20260925 01:30:32.125034 5149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:32.139124 5149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:32.140035 5149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:32.140344 5149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:32.158608 5149 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:40107
I20260925 01:30:32.158627 5214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:40107 every 8 connection(s)
I20260925 01:30:32.160027 5149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:30:32.163180 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 5149
I20260925 01:30:32.163375 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:30:32.164222 28288 external_mini_cluster.cc:1428] Setting key a3443e396477fc4c7d9f01e38160cb75
I20260925 01:30:32.164392 5215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:32.178013 5215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805: Bootstrap starting.
I20260925 01:30:32.181041 5215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:32.182072 5215 log.cc:824] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:32.184773 5215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805: No bootstrap required, opened a new log
I20260925 01:30:32.188506 5218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:32.166409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.27.160.62:40107)
0925 01:30:32.167387 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:32.167400 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:32.167443 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:30:32.169056 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:32.169078 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:32.169607 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:32.169944 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:32.170651 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:32.171952 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:32.173836 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:32.174757 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:32.174897 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:32.185953 (+ 11056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:32.185962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:32.185977 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:32.186573 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:32.186663 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:32.186694 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:32.186800 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:32.187110 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:30:32.187541 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":358,"threads_started":1}
I20260925 01:30:32.192987 5215 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f279061ad24cf7b32fb65f22400805" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40107 } }
I20260925 01:30:32.193404 5215 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:32.193534 5215 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f279061ad24cf7b32fb65f22400805, State: Initialized, Role: FOLLOWER
I20260925 01:30:32.194114 5215 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f279061ad24cf7b32fb65f22400805" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40107 } }
I20260925 01:30:32.194337 5215 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:30:32.194464 5215 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:30:32.194649 5215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:32.197279 5215 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f279061ad24cf7b32fb65f22400805" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40107 } }
I20260925 01:30:32.197777 5215 leader_election.cc:302] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f279061ad24cf7b32fb65f22400805; no voters:
I20260925 01:30:32.198359 5215 leader_election.cc:288] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:30:32.198531 5220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:32.199324 5220 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [term 1 LEADER]: Becoming Leader. State: Replica: c3f279061ad24cf7b32fb65f22400805, State: Running, Role: LEADER
I20260925 01:30:32.199872 5220 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f279061ad24cf7b32fb65f22400805" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40107 } }
I20260925 01:30:32.200572 5215 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:30:32.202858 5222 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f279061ad24cf7b32fb65f22400805. Latest consensus state: current_term: 1 leader_uuid: "c3f279061ad24cf7b32fb65f22400805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f279061ad24cf7b32fb65f22400805" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40107 } } }
I20260925 01:30:32.203228 5222 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:32.206229 5226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:30:32.205965 5221 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f279061ad24cf7b32fb65f22400805" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f279061ad24cf7b32fb65f22400805" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40107 } } }
I20260925 01:30:32.206480 5221 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805 [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:32.211002 5226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:30:32.220544 5226 catalog_manager.cc:1347] Generated new cluster ID: e5e8c4cc25364a7b84337472398a1a03
I20260925 01:30:32.220685 5226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:30:32.231552 5226 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:30:32.232831 5226 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:30:32.241276 5226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f279061ad24cf7b32fb65f22400805: Generated new TSK 0
I20260925 01:30:32.242158 5226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:30:32.252471 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40107
--builtin_ntp_servers=127.27.160.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 01:30:32.610011 5239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:32.610339 5239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:32.610479 5239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:32.617684 5239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:32.617786 5239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:32.617889 5239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:30:32.627254 5239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:32.628590 5239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:32.630527 5239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:32.645689 5244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:32.645773 5247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:32.645650 5245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:32.646749 5239 server_base.cc:1034] running on GCE node
I20260925 01:30:32.647538 5239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:32.648581 5239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:32.649819 5239 hybrid_clock.cc:648] HybridClock initialized: now 1790299832649767 us; error 47 us; skew 500 ppm
I20260925 01:30:32.650164 5239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:30:32.658680 5239 webserver.cc:460] Webserver started at http://127.27.160.1:44467/ using document root <none> and password file <none>
I20260925 01:30:32.659530 5239 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:32.659695 5239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:32.660071 5239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:32.663298 5239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "27268416bfa1432592d0458720f03c76"
format_stamp: "Formatted at 2026-09-25 01:30:32 on dist-test-slave-h6hf"
server_key: "2fe20aee9fc31444a12a9c9a77606ca1"
server_key_iv: "37c4174f9fcb5d8b1ebc717ce2328d09"
server_key_version: "encryptionkey@0"
I20260925 01:30:32.664165 5239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "27268416bfa1432592d0458720f03c76"
format_stamp: "Formatted at 2026-09-25 01:30:32 on dist-test-slave-h6hf"
server_key: "2fe20aee9fc31444a12a9c9a77606ca1"
server_key_iv: "37c4174f9fcb5d8b1ebc717ce2328d09"
server_key_version: "encryptionkey@0"
I20260925 01:30:32.669726 5239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:30:32.673594 5253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:32.675024 5239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:30:32.675212 5239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "27268416bfa1432592d0458720f03c76"
format_stamp: "Formatted at 2026-09-25 01:30:32 on dist-test-slave-h6hf"
server_key: "2fe20aee9fc31444a12a9c9a77606ca1"
server_key_iv: "37c4174f9fcb5d8b1ebc717ce2328d09"
server_key_version: "encryptionkey@0"
I20260925 01:30:32.675449 5239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:32.698196 5239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:32.699096 5239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:32.699409 5239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:32.700451 5239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:32.702126 5239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:32.702241 5239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:32.702373 5239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:32.702459 5239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:32.736368 5239 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:39983
I20260925 01:30:32.736461 5365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:39983 every 8 connection(s)
I20260925 01:30:32.737713 5239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:30:32.743032 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 5239
I20260925 01:30:32.743256 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:30:32.743891 28288 external_mini_cluster.cc:1428] Setting key 05c820c4b5e93e6e8b00b6b05d4a468b
I20260925 01:30:32.746600 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40107
--builtin_ntp_servers=127.27.160.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 01:30:32.750754 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:32.740442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:37399 (local address 127.27.160.62:40107)
0925 01:30:32.740916 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:32.740925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:32.742134 (+ 1209us) server_negotiation.cc:407] Connection header received
0925 01:30:32.744146 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:32.744153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:32.744253 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:32.744370 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:32.745675 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:32.746359 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:32.747959 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:32.748356 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:32.748496 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:32.749897 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:32.749902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:32.749905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:32.749922 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:32.749960 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:32.749972 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:32.750072 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:32.750301 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 01:30:32.750397 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":200,"threads_started":1}
I20260925 01:30:32.751720 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:32.740954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40107 (local address 127.27.160.1:37399)
0925 01:30:32.741958 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:32.742019 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:32.743851 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:32.744605 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:32.744613 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:32.744704 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:32.745494 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:32.745521 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:32.746521 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:32.746526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:32.747812 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:32.747823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:32.748583 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:32.748591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:32.748821 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:32.749584 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:30:32.749586 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:32.749624 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:30:32.749701 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:32.750083 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:32.750096 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:32.750110 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 01:30:32.750276 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 01:30:32.750668 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":342,"threads_started":1}
I20260925 01:30:32.753224 5366 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40107
I20260925 01:30:32.753552 5366 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:32.754292 5366 heartbeater.cc:511] Master 127.27.160.62:40107 requested a full tablet report, sending...
I20260925 01:30:32.756304 5180 ts_manager.cc:194] Registered new tserver with Master: 27268416bfa1432592d0458720f03c76 (127.27.160.1:39983)
I20260925 01:30:32.757861 5180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.1:37399
W20260925 01:30:33.073051 5370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:33.073403 5370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:33.073586 5370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:33.080710 5370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:33.080860 5370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:33.081039 5370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:30:33.090457 5370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:33.091795 5370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:33.093636 5370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:33.106925 5378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:33.107024 5376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:33.107033 5375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:33.108170 5370 server_base.cc:1034] running on GCE node
I20260925 01:30:33.108846 5370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:33.109910 5370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:33.111147 5370 hybrid_clock.cc:648] HybridClock initialized: now 1790299833111085 us; error 57 us; skew 500 ppm
I20260925 01:30:33.111490 5370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:30:33.113919 5370 webserver.cc:460] Webserver started at http://127.27.160.2:38785/ using document root <none> and password file <none>
I20260925 01:30:33.114666 5370 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:33.114806 5370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:33.115146 5370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:33.118191 5370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "40295537902a4372ac21c8f44358855c"
format_stamp: "Formatted at 2026-09-25 01:30:33 on dist-test-slave-h6hf"
server_key: "ef2d810880ccf889b7b693ee68928bcd"
server_key_iv: "c1a050ad90af90510de17e6986019abd"
server_key_version: "encryptionkey@0"
I20260925 01:30:33.119020 5370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "40295537902a4372ac21c8f44358855c"
format_stamp: "Formatted at 2026-09-25 01:30:33 on dist-test-slave-h6hf"
server_key: "ef2d810880ccf889b7b693ee68928bcd"
server_key_iv: "c1a050ad90af90510de17e6986019abd"
server_key_version: "encryptionkey@0"
I20260925 01:30:33.124286 5370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:30:33.127817 5384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:33.129292 5370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:33.129475 5370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "40295537902a4372ac21c8f44358855c"
format_stamp: "Formatted at 2026-09-25 01:30:33 on dist-test-slave-h6hf"
server_key: "ef2d810880ccf889b7b693ee68928bcd"
server_key_iv: "c1a050ad90af90510de17e6986019abd"
server_key_version: "encryptionkey@0"
I20260925 01:30:33.129710 5370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:33.151093 5370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:33.151986 5370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:33.152303 5370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:33.153333 5370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:33.154827 5370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:33.154955 5370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:33.155097 5370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:33.155186 5370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:33.199805 5370 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:40613
I20260925 01:30:33.199893 5496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:40613 every 8 connection(s)
I20260925 01:30:33.201259 5370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:30:33.210085 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 5370
I20260925 01:30:33.210253 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:30:33.210673 28288 external_mini_cluster.cc:1428] Setting key c507ab22aae6d2a39d9cb9c442b8a1e7
I20260925 01:30:33.214190 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40107
--builtin_ntp_servers=127.27.160.20:46361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 01:30:33.214844 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:33.204208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:46207 (local address 127.27.160.62:40107)
0925 01:30:33.204386 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:33.204391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:33.205928 (+ 1537us) server_negotiation.cc:407] Connection header received
0925 01:30:33.207620 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:33.207624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:33.207690 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:33.207780 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:33.208796 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.209594 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.211407 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.211929 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.212049 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:33.213919 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:33.213927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:33.213934 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:33.213959 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:33.214002 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:33.214019 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:33.214117 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:33.214332 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 01:30:33.214448 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:30:33.215777 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:33.204752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40107 (local address 127.27.160.2:46207)
0925 01:30:33.205733 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:33.205770 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:33.207370 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:33.207980 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:33.207988 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:33.208072 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:33.208648 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:33.208666 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.209730 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.209734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:33.211216 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:33.211232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.212210 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.212219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:33.212597 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:33.213576 (+ 979us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:30:33.213578 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:33.213621 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:30:33.213706 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:33.214122 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:33.214129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:33.214136 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:30:33.214281 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 01:30:33.214693 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":313,"threads_started":1}
I20260925 01:30:33.217306 5497 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40107
I20260925 01:30:33.217653 5497 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:33.218617 5497 heartbeater.cc:511] Master 127.27.160.62:40107 requested a full tablet report, sending...
I20260925 01:30:33.220186 5180 ts_manager.cc:194] Registered new tserver with Master: 40295537902a4372ac21c8f44358855c (127.27.160.2:40613)
I20260925 01:30:33.221002 5180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.2:46207
W20260925 01:30:33.562083 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:33.562436 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:33.562579 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:33.570153 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:33.570307 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:33.570494 5500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:30:33.580523 5500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:33.581969 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:33.583741 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:33.597010 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:33.597054 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:33.597152 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:33.599009 5500 server_base.cc:1034] running on GCE node
I20260925 01:30:33.599641 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:33.600641 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:33.601887 5500 hybrid_clock.cc:648] HybridClock initialized: now 1790299833601822 us; error 54 us; skew 500 ppm
I20260925 01:30:33.602244 5500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:30:33.604492 5500 webserver.cc:460] Webserver started at http://127.27.160.3:34275/ using document root <none> and password file <none>
I20260925 01:30:33.605332 5500 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:33.605470 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:33.605834 5500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:33.608696 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "bdec0d1faf8f4db4b48e36811cc17611"
format_stamp: "Formatted at 2026-09-25 01:30:33 on dist-test-slave-h6hf"
server_key: "e6c23d6d1de01cbc7dfe9928a071890e"
server_key_iv: "0ab51d2aa09fa4f9a05e6fcece0ed007"
server_key_version: "encryptionkey@0"
I20260925 01:30:33.609556 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "bdec0d1faf8f4db4b48e36811cc17611"
format_stamp: "Formatted at 2026-09-25 01:30:33 on dist-test-slave-h6hf"
server_key: "e6c23d6d1de01cbc7dfe9928a071890e"
server_key_iv: "0ab51d2aa09fa4f9a05e6fcece0ed007"
server_key_version: "encryptionkey@0"
I20260925 01:30:33.614840 5500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:30:33.618394 5514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:33.619807 5500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:30:33.619944 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "bdec0d1faf8f4db4b48e36811cc17611"
format_stamp: "Formatted at 2026-09-25 01:30:33 on dist-test-slave-h6hf"
server_key: "e6c23d6d1de01cbc7dfe9928a071890e"
server_key_iv: "0ab51d2aa09fa4f9a05e6fcece0ed007"
server_key_version: "encryptionkey@0"
I20260925 01:30:33.620147 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:33.652693 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:33.653586 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:33.653846 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:33.654758 5500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:33.656136 5500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:33.656247 5500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:33.656334 5500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:33.656389 5500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:33.689291 5500 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:33221
I20260925 01:30:33.689368 5626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:33221 every 8 connection(s)
I20260925 01:30:33.690768 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:30:33.699630 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 5500
I20260925 01:30:33.699765 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:30:33.700171 28288 external_mini_cluster.cc:1428] Setting key cce8174737ca369657d4b3028a5ba324
I20260925 01:30:33.705478 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:33.693418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:41951 (local address 127.27.160.62:40107)
0925 01:30:33.693600 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:33.693605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:33.695032 (+ 1427us) server_negotiation.cc:407] Connection header received
0925 01:30:33.696733 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:33.696736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:33.696809 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:33.696899 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:33.698177 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.698914 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.702498 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.702945 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.703091 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:33.704679 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:33.704684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:33.704687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:33.704706 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:33.704741 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:33.704753 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:33.704829 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:33.705043 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 01:30:33.705176 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:30:33.706676 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:33.693969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40107 (local address 127.27.160.3:41951)
0925 01:30:33.694850 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:33.694892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:33.696481 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:33.697195 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:33.697207 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:33.697340 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:33.698010 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:33.698037 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.699063 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.699067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:33.702344 (+ 3277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:33.702358 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.703369 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.703374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:33.703594 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:33.704359 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:30:33.704361 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:33.704406 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:30:33.704490 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:33.704861 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:33.704869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:33.704876 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:30:33.705033 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 01:30:33.705484 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":251,"threads_started":1}
I20260925 01:30:33.708010 5627 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40107
I20260925 01:30:33.708307 5627 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:33.709051 5627 heartbeater.cc:511] Master 127.27.160.62:40107 requested a full tablet report, sending...
I20260925 01:30:33.710518 5180 ts_manager.cc:194] Registered new tserver with Master: bdec0d1faf8f4db4b48e36811cc17611 (127.27.160.3:33221)
I20260925 01:30:33.711380 5180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.3:41951
I20260925 01:30:33.715487 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 01:30:33.721699 5368 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:33.720024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.27.160.62:40107)
0925 01:30:33.720191 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:33.720195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:33.720748 (+ 553us) server_negotiation.cc:407] Connection header received
0925 01:30:33.720921 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:33.720924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:33.720998 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:33.721082 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:33.721529 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33698: BlockingRecv error: recv got EOF from 127.0.0.1:33698 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:30:33.722787 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 5239
I20260925 01:30:33.730881 5636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:33.723743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.27.160.1:39983)
0925 01:30:33.724330 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:33.724336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:33.724363 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:30:33.724418 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:33.724424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:33.724629 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:33.724754 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:33.725256 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.726243 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.727700 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.728323 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.728419 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:33.729489 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:33.729496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:33.729509 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:33.730133 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:33.730179 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:33.730198 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:33.730282 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:33.730402 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 01:30:33.730538 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":297,"threads_started":1}
I20260925 01:30:33.732124 5361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:33.813341 5366 heartbeater.cc:503] Master 127.27.160.62:40107 was elected leader, sending a full tablet report...
I20260925 01:30:33.814564 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 5239
W20260925 01:30:33.830788 5167 connection.cc:535] server connection from 127.27.160.1:37399 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 01:30:33.831126 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 5370
I20260925 01:30:33.839164 5638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:33.832050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.27.160.2:40613)
0925 01:30:33.832610 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:33.832616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:33.832643 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:30:33.832702 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:33.832709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:33.832930 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:33.833129 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:33.833580 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.834712 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.835898 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.836444 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.836543 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:33.837705 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:33.837713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:33.837727 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:33.838339 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:33.838388 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:33.838407 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:33.838499 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:33.838653 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 01:30:33.838794 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":289,"threads_started":1}
I20260925 01:30:33.840425 5492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:33.922281 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 5370
I20260925 01:30:33.938661 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 5500
I20260925 01:30:33.947216 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:33.939724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.27.160.3:33221)
0925 01:30:33.940355 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:33.940362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:33.940390 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:30:33.940449 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:33.940455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:33.940673 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:33.940860 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:33.941390 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.942420 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.943643 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:33.944233 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:33.944377 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:33.945704 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:33.945714 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:33.945727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:33.946335 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:33.946394 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:33.946413 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:33.946545 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:33.946702 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 01:30:33.946847 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":340,"threads_started":1}
I20260925 01:30:33.948486 5622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:34.030989 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 5500
I20260925 01:30:34.046733 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 5149
I20260925 01:30:34.047935 5210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:34.150938 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 5149
2026-09-25T01:30:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2480 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T01:30:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:30:34Z Disabled control of system clock
I20260925 01:30:34.223140 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44631
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:44631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 01:30:34.560670 5646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:34.560985 5646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:34.561136 5646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:34.568537 5646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:30:34.568660 5646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:34.568697 5646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:34.568747 5646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:30:34.568841 5646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:30:34.578313 5646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:44631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:44631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:34.579690 5646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:34.581584 5646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:34.593832 5651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:34.593899 5652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:34.593860 5654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:34.594395 5646 server_base.cc:1034] running on GCE node
I20260925 01:30:34.595371 5646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:34.596774 5646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:34.598011 5646 hybrid_clock.cc:648] HybridClock initialized: now 1790299834597976 us; error 50 us; skew 500 ppm
I20260925 01:30:34.598389 5646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:30:34.600620 5646 webserver.cc:460] Webserver started at http://127.27.160.62:35801/ using document root <none> and password file <none>
I20260925 01:30:34.601449 5646 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:34.601595 5646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:34.601924 5646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:34.604820 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "c08201e0216849968fac9a451db41838"
format_stamp: "Formatted at 2026-09-25 01:30:34 on dist-test-slave-h6hf"
server_key: "23d970c11d2e1141662a18b431d07335"
server_key_iv: "6632eb5061ab7c9189dc7e009cc4c3c1"
server_key_version: "encryptionkey@0"
I20260925 01:30:34.605698 5646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "c08201e0216849968fac9a451db41838"
format_stamp: "Formatted at 2026-09-25 01:30:34 on dist-test-slave-h6hf"
server_key: "23d970c11d2e1141662a18b431d07335"
server_key_iv: "6632eb5061ab7c9189dc7e009cc4c3c1"
server_key_version: "encryptionkey@0"
I20260925 01:30:34.610882 5646 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:30:34.614379 5660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:34.615707 5646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:30:34.615886 5646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "c08201e0216849968fac9a451db41838"
format_stamp: "Formatted at 2026-09-25 01:30:34 on dist-test-slave-h6hf"
server_key: "23d970c11d2e1141662a18b431d07335"
server_key_iv: "6632eb5061ab7c9189dc7e009cc4c3c1"
server_key_version: "encryptionkey@0"
I20260925 01:30:34.616111 5646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:34.621973 5646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:34.622762 5646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:34.623071 5646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:34.641604 5646 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:44631
I20260925 01:30:34.641618 5711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:44631 every 8 connection(s)
I20260925 01:30:34.643069 5646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:30:34.645407 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 5646
I20260925 01:30:34.645656 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:30:34.646472 28288 external_mini_cluster.cc:1428] Setting key 09f35aeb37043b6b4c00329e1bfa591f
I20260925 01:30:34.649061 5712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:34.655088 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:34.648494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.27.160.62:44631)
0925 01:30:34.649513 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:34.649526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:34.649569 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:30:34.651132 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:34.651154 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:34.651595 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:34.652032 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:34.653265 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:34.653272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:34.653285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:34.653303 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:34.653384 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:34.653401 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:34.653675 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:34.653863 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 01:30:34.654127 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":302,"threads_started":1}
I20260925 01:30:34.660997 5712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838: Bootstrap starting.
I20260925 01:30:34.664572 5712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:34.665740 5712 log.cc:824] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:34.668771 5712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838: No bootstrap required, opened a new log
I20260925 01:30:34.676553 5712 raft_consensus.cc:348] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08201e0216849968fac9a451db41838" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44631 } }
I20260925 01:30:34.676918 5712 raft_consensus.cc:374] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:34.677034 5712 raft_consensus.cc:729] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08201e0216849968fac9a451db41838, State: Initialized, Role: FOLLOWER
I20260925 01:30:34.677686 5712 consensus_queue.cc:260] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08201e0216849968fac9a451db41838" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44631 } }
I20260925 01:30:34.677942 5712 raft_consensus.cc:388] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:30:34.678094 5712 raft_consensus.cc:482] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:30:34.678287 5712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:34.680958 5712 raft_consensus.cc:504] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08201e0216849968fac9a451db41838" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44631 } }
I20260925 01:30:34.681506 5712 leader_election.cc:302] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c08201e0216849968fac9a451db41838; no voters:
I20260925 01:30:34.682123 5712 leader_election.cc:288] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:30:34.682261 5717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:34.682964 5717 raft_consensus.cc:686] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [term 1 LEADER]: Becoming Leader. State: Replica: c08201e0216849968fac9a451db41838, State: Running, Role: LEADER
I20260925 01:30:34.683555 5717 consensus_queue.cc:237] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08201e0216849968fac9a451db41838" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44631 } }
I20260925 01:30:34.684296 5712 sys_catalog.cc:564] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:30:34.687870 5719 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c08201e0216849968fac9a451db41838. Latest consensus state: current_term: 1 leader_uuid: "c08201e0216849968fac9a451db41838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08201e0216849968fac9a451db41838" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44631 } } }
I20260925 01:30:34.688364 5719 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:34.689028 5718 sys_catalog.cc:455] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c08201e0216849968fac9a451db41838" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c08201e0216849968fac9a451db41838" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 44631 } } }
I20260925 01:30:34.689487 5718 sys_catalog.cc:458] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838 [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:34.690553 5726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:30:34.694893 5726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:30:34.703524 5726 catalog_manager.cc:1347] Generated new cluster ID: 58898f338bf8464aabbaef6ed23dd7c9
I20260925 01:30:34.703655 5726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:30:34.731369 5726 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:30:34.731846 5726 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:30:34.741576 5726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c08201e0216849968fac9a451db41838: Generated new TSK 0
I20260925 01:30:34.742426 5726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:30:34.748322 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44631
--builtin_ntp_servers=127.27.160.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 01:30:35.082547 5736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:35.082907 5736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:35.083086 5736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:35.090982 5736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:35.091125 5736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:35.091307 5736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:30:35.100705 5736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:35.102162 5736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:35.104081 5736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:35.118103 5742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:35.118057 5741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:35.118140 5744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:35.118805 5736 server_base.cc:1034] running on GCE node
I20260925 01:30:35.119515 5736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:35.120664 5736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:35.121862 5736 hybrid_clock.cc:648] HybridClock initialized: now 1790299835121787 us; error 69 us; skew 500 ppm
I20260925 01:30:35.122207 5736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:30:35.124480 5736 webserver.cc:460] Webserver started at http://127.27.160.1:45311/ using document root <none> and password file <none>
I20260925 01:30:35.125296 5736 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:35.125418 5736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:35.125754 5736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:35.130592 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "99f022dbad2d4779a295961a17946fb9"
format_stamp: "Formatted at 2026-09-25 01:30:35 on dist-test-slave-h6hf"
server_key: "2ec9ad19f5f84085cc46591741d1159a"
server_key_iv: "e341bc80e5e44b3fc5229fb998c3a6a8"
server_key_version: "encryptionkey@0"
I20260925 01:30:35.131448 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "99f022dbad2d4779a295961a17946fb9"
format_stamp: "Formatted at 2026-09-25 01:30:35 on dist-test-slave-h6hf"
server_key: "2ec9ad19f5f84085cc46591741d1159a"
server_key_iv: "e341bc80e5e44b3fc5229fb998c3a6a8"
server_key_version: "encryptionkey@0"
I20260925 01:30:35.137066 5736 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:30:35.140790 5750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:35.142338 5736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:35.142525 5736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "99f022dbad2d4779a295961a17946fb9"
format_stamp: "Formatted at 2026-09-25 01:30:35 on dist-test-slave-h6hf"
server_key: "2ec9ad19f5f84085cc46591741d1159a"
server_key_iv: "e341bc80e5e44b3fc5229fb998c3a6a8"
server_key_version: "encryptionkey@0"
I20260925 01:30:35.142763 5736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:35.148550 5736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:35.149459 5736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:35.149741 5736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:35.150656 5736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:35.152025 5736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:35.152148 5736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:35.152276 5736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:35.152366 5736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:35.185325 5736 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:40419
I20260925 01:30:35.185343 5862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:40419 every 8 connection(s)
I20260925 01:30:35.186718 5736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:30:35.191856 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 5736
I20260925 01:30:35.192108 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:30:35.192898 28288 external_mini_cluster.cc:1428] Setting key 04e38733dfd26aafe66c733d6bfb3fb0
I20260925 01:30:35.196141 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:35.189396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:54063 (local address 127.27.160.62:44631)
0925 01:30:35.189903 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:35.189907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:35.191167 (+ 1260us) server_negotiation.cc:407] Connection header received
0925 01:30:35.193727 (+ 2560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:35.193733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:35.193806 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:35.193937 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:35.195421 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:35.195425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:35.195428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:35.195444 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:35.195486 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:35.195497 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:35.195662 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:35.195884 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 01:30:35.195934 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":315,"threads_started":1}
I20260925 01:30:35.196141 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44631
--builtin_ntp_servers=127.27.160.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 01:30:35.197628 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:35.190077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44631 (local address 127.27.160.1:54063)
0925 01:30:35.191011 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:35.191051 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:35.193368 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:35.194208 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:35.194217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:35.194318 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:35.195071 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:30:35.195075 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:35.195126 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:30:35.195243 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:35.195645 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:35.195657 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:35.195665 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:30:35.195974 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 01:30:35.196264 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":301,"threads_started":1}
I20260925 01:30:35.199178 5863 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44631
I20260925 01:30:35.199507 5863 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:35.200201 5863 heartbeater.cc:511] Master 127.27.160.62:44631 requested a full tablet report, sending...
I20260925 01:30:35.202411 5677 ts_manager.cc:194] Registered new tserver with Master: 99f022dbad2d4779a295961a17946fb9 (127.27.160.1:40419)
W20260925 01:30:35.539420 5867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:35.539980 5867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:35.540264 5867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:35.549908 5867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:35.550014 5867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:35.550135 5867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:30:35.560530 5867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:35.561959 5867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:35.564095 5867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:35.577919 5872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:35.578029 5875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:35.578089 5873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:35.579207 5867 server_base.cc:1034] running on GCE node
I20260925 01:30:35.579931 5867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:35.580981 5867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:35.582227 5867 hybrid_clock.cc:648] HybridClock initialized: now 1790299835582182 us; error 63 us; skew 500 ppm
I20260925 01:30:35.582669 5867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:30:35.585713 5867 webserver.cc:460] Webserver started at http://127.27.160.2:39255/ using document root <none> and password file <none>
I20260925 01:30:35.586437 5867 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:35.586582 5867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:35.586911 5867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:35.589919 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "33ed62858d5547cc9c748c97086efcf5"
format_stamp: "Formatted at 2026-09-25 01:30:35 on dist-test-slave-h6hf"
server_key: "9343878305314b9e2393c0ba78ddbcd4"
server_key_iv: "189d03ada697d1382175fa2612ea11f5"
server_key_version: "encryptionkey@0"
I20260925 01:30:35.590720 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "33ed62858d5547cc9c748c97086efcf5"
format_stamp: "Formatted at 2026-09-25 01:30:35 on dist-test-slave-h6hf"
server_key: "9343878305314b9e2393c0ba78ddbcd4"
server_key_iv: "189d03ada697d1382175fa2612ea11f5"
server_key_version: "encryptionkey@0"
I20260925 01:30:35.596130 5867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 01:30:35.599581 5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:35.601073 5867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:30:35.601292 5867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "33ed62858d5547cc9c748c97086efcf5"
format_stamp: "Formatted at 2026-09-25 01:30:35 on dist-test-slave-h6hf"
server_key: "9343878305314b9e2393c0ba78ddbcd4"
server_key_iv: "189d03ada697d1382175fa2612ea11f5"
server_key_version: "encryptionkey@0"
I20260925 01:30:35.601521 5867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:35.607208 5867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:35.608012 5867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:35.608274 5867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:35.609206 5867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:35.610611 5867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:35.610721 5867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:35.610852 5867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:35.610944 5867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:35.642683 5867 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:42887
I20260925 01:30:35.642779 5993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:42887 every 8 connection(s)
I20260925 01:30:35.644016 5867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:30:35.651054 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 5867
I20260925 01:30:35.651265 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:30:35.651803 28288 external_mini_cluster.cc:1428] Setting key b969ada92f1b61b409b9ea9052f796fe
I20260925 01:30:35.653843 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:35.646896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:46403 (local address 127.27.160.62:44631)
0925 01:30:35.647130 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:35.647137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:35.648750 (+ 1613us) server_negotiation.cc:407] Connection header received
0925 01:30:35.651282 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:35.651290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:35.651363 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:35.651488 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:35.653035 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:35.653041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:35.653046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:35.653071 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:35.653107 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:35.653167 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:35.653276 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:35.653479 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 01:30:35.653568 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:30:35.654824 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:35.647532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44631 (local address 127.27.160.2:46403)
0925 01:30:35.648577 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:35.648629 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:35.650901 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:35.651735 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:35.651747 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:35.651869 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:35.652679 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:30:35.652682 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:35.652738 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:30:35.652863 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:35.653249 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:35.653260 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:35.653268 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:30:35.653583 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 01:30:35.653859 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":288,"threads_started":1}
I20260925 01:30:35.655098 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44631
--builtin_ntp_servers=127.27.160.20:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 01:30:35.658170 5994 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44631
I20260925 01:30:35.658527 5994 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:35.659135 5994 heartbeater.cc:511] Master 127.27.160.62:44631 requested a full tablet report, sending...
I20260925 01:30:35.660607 5677 ts_manager.cc:194] Registered new tserver with Master: 33ed62858d5547cc9c748c97086efcf5 (127.27.160.2:42887)
W20260925 01:30:35.986047 5997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:35.986330 5997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:35.986419 5997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:35.993863 5997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:35.993965 5997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:35.994076 5997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:30:36.003304 5997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:44631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:36.004740 5997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:36.006807 5997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:36.021373 6005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:36.021445 6003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:36.021462 6002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:36.027773 5997 server_base.cc:1034] running on GCE node
I20260925 01:30:36.028541 5997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:36.029654 5997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:36.030891 5997 hybrid_clock.cc:648] HybridClock initialized: now 1790299836030833 us; error 55 us; skew 500 ppm
I20260925 01:30:36.031231 5997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:30:36.033609 5997 webserver.cc:460] Webserver started at http://127.27.160.3:40753/ using document root <none> and password file <none>
I20260925 01:30:36.034319 5997 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:36.034453 5997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:36.034787 5997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:36.037726 5997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "de072ee33cda48e1967d927075aa782a"
format_stamp: "Formatted at 2026-09-25 01:30:36 on dist-test-slave-h6hf"
server_key: "5a2e6e50cc980fea79c029697cf50222"
server_key_iv: "2f3efcc9395e09e3224f7c73219eb30f"
server_key_version: "encryptionkey@0"
I20260925 01:30:36.038506 5997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "de072ee33cda48e1967d927075aa782a"
format_stamp: "Formatted at 2026-09-25 01:30:36 on dist-test-slave-h6hf"
server_key: "5a2e6e50cc980fea79c029697cf50222"
server_key_iv: "2f3efcc9395e09e3224f7c73219eb30f"
server_key_version: "encryptionkey@0"
I20260925 01:30:36.043630 5997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:30:36.047257 6011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:36.048676 5997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:30:36.048856 5997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "de072ee33cda48e1967d927075aa782a"
format_stamp: "Formatted at 2026-09-25 01:30:36 on dist-test-slave-h6hf"
server_key: "5a2e6e50cc980fea79c029697cf50222"
server_key_iv: "2f3efcc9395e09e3224f7c73219eb30f"
server_key_version: "encryptionkey@0"
I20260925 01:30:36.049090 5997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:36.055068 5997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:36.055900 5997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:36.056171 5997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:36.057104 5997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:36.058629 5997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:36.058754 5997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:36.058892 5997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:36.058979 5997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:36.092459 5997 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:35541
I20260925 01:30:36.092494 6123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:35541 every 8 connection(s)
I20260925 01:30:36.093967 5997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:30:36.101365 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 5997
I20260925 01:30:36.101591 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:30:36.102140 28288 external_mini_cluster.cc:1428] Setting key 7004447ae6b225c053ea034356df2808
I20260925 01:30:36.102277 5865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:36.096655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:48403 (local address 127.27.160.62:44631)
0925 01:30:36.096831 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:36.096836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:36.098351 (+ 1515us) server_negotiation.cc:407] Connection header received
0925 01:30:36.099994 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:36.099998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:36.100052 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:36.100141 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:36.101485 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:36.101492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:36.101498 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:36.101520 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:36.101563 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:36.101584 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:36.101711 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:36.101915 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 01:30:36.101980 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:30:36.103277 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:36.097269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:44631 (local address 127.27.160.3:48403)
0925 01:30:36.098190 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:36.098232 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:36.099729 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:36.100345 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:36.100353 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:36.100446 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:36.101171 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:30:36.101180 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:36.101221 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:30:36.101303 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:36.101706 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:36.101716 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:36.101730 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 01:30:36.101949 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 01:30:36.102158 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":279,"threads_started":1}
I20260925 01:30:36.104944 6124 heartbeater.cc:347] Connected to a master server at 127.27.160.62:44631
I20260925 01:30:36.105291 6124 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:36.105932 6124 heartbeater.cc:511] Master 127.27.160.62:44631 requested a full tablet report, sending...
I20260925 01:30:36.107321 5677 ts_manager.cc:194] Registered new tserver with Master: de072ee33cda48e1967d927075aa782a (127.27.160.3:35541)
I20260925 01:30:36.117797 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 01:30:36.123950 5865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:30:36.122461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.27.160.62:44631)
0925 01:30:36.122642 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:36.122647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:36.123142 (+ 495us) server_negotiation.cc:407] Connection header received
0925 01:30:36.123278 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:36.123282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:36.123331 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:36.123416 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:36.123682 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58816: BlockingRecv error: recv got EOF from 127.0.0.1:58816 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:30:36.125101 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 5736
I20260925 01:30:36.128562 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:36.126118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.27.160.1:40419)
0925 01:30:36.126651 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:36.126658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:36.126685 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:30:36.126744 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:36.126750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:36.126969 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:36.127097 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:36.127958 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:36.127965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:36.127979 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:36.127996 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:36.128037 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:36.128050 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:36.128153 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:36.128237 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 01:30:36.128313 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260925 01:30:36.129841 5858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:36.209750 5863 heartbeater.cc:503] Master 127.27.160.62:44631 was elected leader, sending a full tablet report...
I20260925 01:30:36.211737 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 5736
I20260925 01:30:36.228032 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 5867
I20260925 01:30:36.231576 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:36.229092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.27.160.2:42887)
0925 01:30:36.229615 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:36.229623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:36.229649 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:30:36.229710 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:36.229716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:36.229936 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:36.230075 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:36.230970 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:36.230976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:36.230988 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:36.231010 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:36.231064 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:36.231078 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:36.231180 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:36.231259 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 01:30:36.231338 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":245,"threads_started":1}
I20260925 01:30:36.232774 5989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:36.314957 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 5867
I20260925 01:30:36.331235 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 5997
I20260925 01:30:36.335311 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:36.332161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.27.160.3:35541)
0925 01:30:36.332783 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:36.332790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:36.332820 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:30:36.332880 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:36.332886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:36.333189 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:36.333361 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:36.334556 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:36.334562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:36.334573 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:30:36.334584 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:36.334626 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:30:36.334639 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:36.334823 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:36.334892 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 01:30:36.335012 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":309,"threads_started":1}
I20260925 01:30:36.336535 6119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:36.418044 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 5997
I20260925 01:30:36.433792 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 5646
I20260925 01:30:36.435065 5707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:36.537971 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 5646
2026-09-25T01:30:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2387 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:30:36 dist-test-slave-h6hf krb5kdc[6142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:30:36 dist-test-slave-h6hf krb5kdc[6142](info): set up 2 sockets
Sep 25 01:30:36 dist-test-slave-h6hf krb5kdc[6142](info): commencing operation
krb5kdc: starting...
W20260925 01:30:37.731087 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:30:37 dist-test-slave-h6hf krb5kdc[6142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:30:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:30:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:30:37.969337 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37315
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:38.303553 6155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:38.303872 6155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:38.304054 6155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:38.312340 6155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:30:38.312485 6155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:38.312539 6155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:38.312637 6155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:30:38.312724 6155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:30:38.323333 6155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:37315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:38.324712 6155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:38.326305 6155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:38.338750 6161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:38.338820 6163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:38.338793 6160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:38.344854 6155 server_base.cc:1034] running on GCE node
I20260925 01:30:38.345827 6155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:38.347424 6155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:38.348665 6155 hybrid_clock.cc:648] HybridClock initialized: now 1790299838348631 us; error 46 us; skew 500 ppm
I20260925 01:30:38.349030 6155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:38 dist-test-slave-h6hf krb5kdc[6142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:38.356449 6155 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:30:38.358187 6155 webserver.cc:460] Webserver started at http://127.27.160.62:44233/ using document root <none> and password file <none>
I20260925 01:30:38.358973 6155 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:38.359127 6155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:38.359453 6155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:38.362596 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "2ed944fed3f84d01af8c34e4c76946ad"
format_stamp: "Formatted at 2026-09-25 01:30:38 on dist-test-slave-h6hf"
server_key: "ee4c13cb31889be7b7d7ef8914b01fa1"
server_key_iv: "8849a334616d26e9b9dfb1822b7c9e73"
server_key_version: "encryptionkey@0"
I20260925 01:30:38.363394 6155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "2ed944fed3f84d01af8c34e4c76946ad"
format_stamp: "Formatted at 2026-09-25 01:30:38 on dist-test-slave-h6hf"
server_key: "ee4c13cb31889be7b7d7ef8914b01fa1"
server_key_iv: "8849a334616d26e9b9dfb1822b7c9e73"
server_key_version: "encryptionkey@0"
I20260925 01:30:38.368973 6155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:30:38.372645 6170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:38.374229 6155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:38.374439 6155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "2ed944fed3f84d01af8c34e4c76946ad"
format_stamp: "Formatted at 2026-09-25 01:30:38 on dist-test-slave-h6hf"
server_key: "ee4c13cb31889be7b7d7ef8914b01fa1"
server_key_iv: "8849a334616d26e9b9dfb1822b7c9e73"
server_key_version: "encryptionkey@0"
I20260925 01:30:38.374667 6155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:38.399358 6155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:38.403640 6155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:38.403940 6155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:38.422704 6155 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:37315
I20260925 01:30:38.422729 6221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:37315 every 8 connection(s)
I20260925 01:30:38.424153 6155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:30:38.427682 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 6155
I20260925 01:30:38.427913 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:30:38.427992 6222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:38.428607 28288 external_mini_cluster.cc:1428] Setting key c46639e11ba2b1cd9dfdc5a33e9a358b
I20260925 01:30:38.440423 6222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad: Bootstrap starting.
Sep 25 01:30:38 dist-test-slave-h6hf krb5kdc[6142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:30:38.444108 6222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:38.445439 6222 log.cc:824] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:38.448590 6222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad: No bootstrap required, opened a new log
I20260925 01:30:38.451930 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:38.430699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.27.160.62:37315)
0925 01:30:38.431501 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:38.431512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:38.431553 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:30:38.433043 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:38.433064 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:38.433534 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:38.434001 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:38.435733 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:38.437046 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:38.438407 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:38.439356 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:38.439595 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:38.444550 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:38.444559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:38.444577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:38.444622 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:38.447747 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:38.448451 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:38.448507 (+ 56us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:38.448513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:38.448741 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:38.449159 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:38.449167 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:38.449172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:38.449683 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:38.450035 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:38.450462 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 01:30:38.450896 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":360,"threads_started":1}
I20260925 01:30:38.457304 6222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed944fed3f84d01af8c34e4c76946ad" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37315 } }
I20260925 01:30:38.457702 6222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:38.457841 6222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed944fed3f84d01af8c34e4c76946ad, State: Initialized, Role: FOLLOWER
I20260925 01:30:38.458431 6222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed944fed3f84d01af8c34e4c76946ad" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37315 } }
I20260925 01:30:38.458683 6222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:30:38.458870 6222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:30:38.459056 6222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:38.462029 6222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed944fed3f84d01af8c34e4c76946ad" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37315 } }
I20260925 01:30:38.462584 6222 leader_election.cc:302] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ed944fed3f84d01af8c34e4c76946ad; no voters:
I20260925 01:30:38.463200 6222 leader_election.cc:288] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:30:38.463366 6227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:38.464238 6227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [term 1 LEADER]: Becoming Leader. State: Replica: 2ed944fed3f84d01af8c34e4c76946ad, State: Running, Role: LEADER
I20260925 01:30:38.464761 6227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed944fed3f84d01af8c34e4c76946ad" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37315 } }
I20260925 01:30:38.465691 6222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:30:38.467723 6228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ed944fed3f84d01af8c34e4c76946ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed944fed3f84d01af8c34e4c76946ad" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37315 } } }
I20260925 01:30:38.468151 6229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ed944fed3f84d01af8c34e4c76946ad. Latest consensus state: current_term: 1 leader_uuid: "2ed944fed3f84d01af8c34e4c76946ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed944fed3f84d01af8c34e4c76946ad" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37315 } } }
I20260925 01:30:38.468712 6229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:38.469619 6228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad [sys.catalog]: This master's current role is: LEADER
I20260925 01:30:38.472863 6232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:30:38.478492 6232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:30:38.487752 6232 catalog_manager.cc:1347] Generated new cluster ID: 1eaee15e65ef411186ba72121e9b5fc5
I20260925 01:30:38.487869 6232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:30:38.511354 6232 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:30:38.513386 6232 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:30:38.528827 6232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad: Generated new TSK 0
I20260925 01:30:38.530058 6232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:30:38.597756 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37315
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:38.939839 6248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:38.940171 6248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:38.940336 6248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:38.947973 6248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:38.948132 6248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:38.948318 6248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:30:38.958638 6248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:38.960041 6248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:38.961452 6248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:38.975104 6253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:38.975028 6256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:38.975067 6254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:38.975876 6248 server_base.cc:1034] running on GCE node
I20260925 01:30:38.976507 6248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:38.977535 6248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:38.978780 6248 hybrid_clock.cc:648] HybridClock initialized: now 1790299838978721 us; error 46 us; skew 500 ppm
I20260925 01:30:38.979128 6248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:38 dist-test-slave-h6hf krb5kdc[6142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:38.982371 6248 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:30:38.983785 6248 webserver.cc:460] Webserver started at http://127.27.160.1:33487/ using document root <none> and password file <none>
I20260925 01:30:38.984509 6248 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:38.984658 6248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:38.984974 6248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:38.987903 6248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "be55fa2375d741e4b0370eeb94c4426b"
format_stamp: "Formatted at 2026-09-25 01:30:38 on dist-test-slave-h6hf"
server_key: "c86014c19aa0296f70b39a2e0beee2ae"
server_key_iv: "c07ddad314e2fe461ee211eb0bd8be45"
server_key_version: "encryptionkey@0"
I20260925 01:30:38.988683 6248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "be55fa2375d741e4b0370eeb94c4426b"
format_stamp: "Formatted at 2026-09-25 01:30:38 on dist-test-slave-h6hf"
server_key: "c86014c19aa0296f70b39a2e0beee2ae"
server_key_iv: "c07ddad314e2fe461ee211eb0bd8be45"
server_key_version: "encryptionkey@0"
I20260925 01:30:38.994021 6248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:30:38.997334 6263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:38.998626 6248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:30:38.998811 6248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "be55fa2375d741e4b0370eeb94c4426b"
format_stamp: "Formatted at 2026-09-25 01:30:38 on dist-test-slave-h6hf"
server_key: "c86014c19aa0296f70b39a2e0beee2ae"
server_key_iv: "c07ddad314e2fe461ee211eb0bd8be45"
server_key_version: "encryptionkey@0"
I20260925 01:30:38.999019 6248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:39.011611 6248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:39.016348 6248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:39.016672 6248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:39.017879 6248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:39.019292 6248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:39.019404 6248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:39.019555 6248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:39.019629 6248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:39.052819 6248 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:42657
I20260925 01:30:39.052865 6375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:42657 every 8 connection(s)
I20260925 01:30:39.054276 6248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:30:39.064695 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 6248
I20260925 01:30:39.064944 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:30:39.065644 28288 external_mini_cluster.cc:1428] Setting key e24a3eebb08a03455a99b00421c4c884
Sep 25 01:30:39 dist-test-slave-h6hf krb5kdc[6142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:30:39.075804 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:39.056849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:55547 (local address 127.27.160.62:37315)
0925 01:30:39.057356 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:39.057361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:39.058263 (+ 902us) server_negotiation.cc:407] Connection header received
0925 01:30:39.060146 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:39.060152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:39.060245 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:39.060366 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:39.061890 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:39.062611 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:39.064191 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:39.064593 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:39.064728 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:39.069321 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:39.069326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:39.069329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:39.069359 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:39.072028 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:39.073106 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:39.073110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:39.073112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:39.073280 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:39.073932 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:39.073936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:39.073938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:39.074203 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:39.074314 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:39.075361 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0925 01:30:39.075543 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":283,"threads_started":1}
I20260925 01:30:39.077468 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:39.057360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37315 (local address 127.27.160.1:55547)
0925 01:30:39.058082 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:39.058127 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:39.059850 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:39.060604 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:39.060613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:39.061141 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:39.061704 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:39.061721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:39.062767 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:39.062773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:39.064031 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:39.064044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:39.064827 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:39.064837 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:39.065181 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:39.065934 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:39.065956 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:39.069059 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:39.072190 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:39.072203 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:39.072226 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:39.072975 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:39.073433 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:39.073442 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:39.073449 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:39.073780 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:39.074348 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:39.074360 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:39.074884 (+ 524us) client_negotiation.cc:769] Sending connection context
0925 01:30:39.075271 (+ 387us) client_negotiation.cc:240] Negotiation successful
0925 01:30:39.075935 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":271,"threads_started":1}
I20260925 01:30:39.079277 6376 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37315
I20260925 01:30:39.079669 6376 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:39.080528 6376 heartbeater.cc:511] Master 127.27.160.62:37315 requested a full tablet report, sending...
I20260925 01:30:39.083042 6187 ts_manager.cc:194] Registered new tserver with Master: be55fa2375d741e4b0370eeb94c4426b (127.27.160.1:42657)
I20260925 01:30:39.085237 6187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:55547
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:30:39.120949 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37315
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:39.453377 6382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:39.453694 6382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:39.453792 6382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:39.461555 6382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:39.461695 6382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:39.461839 6382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:30:39.472673 6382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:39.474185 6382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:39.475474 6382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:39.489248 6390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:39.489351 6387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:39.489305 6388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:39.489959 6382 server_base.cc:1034] running on GCE node
I20260925 01:30:39.490599 6382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:39.491575 6382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:39.492818 6382 hybrid_clock.cc:648] HybridClock initialized: now 1790299839492755 us; error 55 us; skew 500 ppm
I20260925 01:30:39.493232 6382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:39 dist-test-slave-h6hf krb5kdc[6142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299839, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:39.496407 6382 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:30:39.498009 6382 webserver.cc:460] Webserver started at http://127.27.160.2:43177/ using document root <none> and password file <none>
I20260925 01:30:39.498747 6382 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:39.498903 6382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:39.499219 6382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:39.502292 6382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "1a54b1706e01464e9f9e303ae2a7ceb3"
format_stamp: "Formatted at 2026-09-25 01:30:39 on dist-test-slave-h6hf"
server_key: "2150fb882d90c6119dfaca708ac60749"
server_key_iv: "38435d507ebf1f179b2ca87a91309af0"
server_key_version: "encryptionkey@0"
I20260925 01:30:39.503187 6382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "1a54b1706e01464e9f9e303ae2a7ceb3"
format_stamp: "Formatted at 2026-09-25 01:30:39 on dist-test-slave-h6hf"
server_key: "2150fb882d90c6119dfaca708ac60749"
server_key_iv: "38435d507ebf1f179b2ca87a91309af0"
server_key_version: "encryptionkey@0"
I20260925 01:30:39.508800 6382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:30:39.512428 6397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:39.514038 6382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:30:39.514240 6382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "1a54b1706e01464e9f9e303ae2a7ceb3"
format_stamp: "Formatted at 2026-09-25 01:30:39 on dist-test-slave-h6hf"
server_key: "2150fb882d90c6119dfaca708ac60749"
server_key_iv: "38435d507ebf1f179b2ca87a91309af0"
server_key_version: "encryptionkey@0"
I20260925 01:30:39.514495 6382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:39.529373 6382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:39.535892 6382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:39.536247 6382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:39.537628 6382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:39.539672 6382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:39.539789 6382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:39.539947 6382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:39.540036 6382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:39.575393 6382 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:41007
I20260925 01:30:39.575433 6509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:41007 every 8 connection(s)
I20260925 01:30:39.576862 6382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:30:39.585670 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 6382
I20260925 01:30:39.585877 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:30:39.586339 28288 external_mini_cluster.cc:1428] Setting key 0b7ad1a207baec3bb7d0e05aa0ec2d63
Sep 25 01:30:39 dist-test-slave-h6hf krb5kdc[6142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299839, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:30:39.598372 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:39.579778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:41297 (local address 127.27.160.62:37315)
0925 01:30:39.580239 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:39.580245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:39.581328 (+ 1083us) server_negotiation.cc:407] Connection header received
0925 01:30:39.583091 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:39.583095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:39.583169 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:39.583271 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:39.584869 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:39.585784 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:39.587639 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:39.588465 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:39.588615 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:39.592241 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:39.592246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:39.592250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:39.592283 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:39.595009 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:39.595855 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:39.595861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:39.595864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:39.596027 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:39.596562 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:39.596566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:39.596568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:39.596837 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:39.596969 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:39.597915 (+ 946us) server_negotiation.cc:299] Negotiation successful
0925 01:30:39.598109 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260925 01:30:39.599992 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:39.580404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37315 (local address 127.27.160.2:41297)
0925 01:30:39.581160 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:39.581219 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:39.582818 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:39.583483 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:39.583491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:39.584028 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:39.584675 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:39.584696 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:39.585984 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:39.585993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:39.587490 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:39.587502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:39.588711 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:39.588719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:39.588968 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:39.589758 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:39.589783 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:39.592002 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:39.595169 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:39.595184 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:39.595203 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:39.595719 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:39.596154 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:39.596161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:39.596167 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:39.596418 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:39.596973 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:39.596985 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:39.597526 (+ 541us) client_negotiation.cc:769] Sending connection context
0925 01:30:39.597844 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 01:30:39.598457 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":276,"threads_started":1}
I20260925 01:30:39.601604 6510 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37315
I20260925 01:30:39.601929 6510 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:39.602795 6510 heartbeater.cc:511] Master 127.27.160.62:37315 requested a full tablet report, sending...
I20260925 01:30:39.604475 6187 ts_manager.cc:194] Registered new tserver with Master: 1a54b1706e01464e9f9e303ae2a7ceb3 (127.27.160.2:41007)
I20260925 01:30:39.605798 6187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:41297
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:30:39.640740 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37315
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:46807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:30:39.972993 6516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:30:39.973371 6516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:30:39.973583 6516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:30:39.981186 6516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:30:39.981323 6516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:30:39.981500 6516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:30:39.991248 6516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37315
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:30:39.992565 6516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:30:39.993996 6516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:30:40.007511 6522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:40.007479 6521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:30:40.007648 6524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:30:40.008196 6516 server_base.cc:1034] running on GCE node
I20260925 01:30:40.008922 6516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:30:40.009914 6516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:30:40.011113 6516 hybrid_clock.cc:648] HybridClock initialized: now 1790299840011078 us; error 34 us; skew 500 ppm
I20260925 01:30:40.011433 6516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:30:40 dist-test-slave-h6hf krb5kdc[6142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299840, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:30:40.014796 6516 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:30:40.016422 6516 webserver.cc:460] Webserver started at http://127.27.160.3:44365/ using document root <none> and password file <none>
I20260925 01:30:40.017238 6516 fs_manager.cc:359] Metadata directory not provided
I20260925 01:30:40.017349 6516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:30:40.017616 6516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:30:40.020790 6516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "5d9edb25d92d4ae69ed3470a53935700"
format_stamp: "Formatted at 2026-09-25 01:30:40 on dist-test-slave-h6hf"
server_key: "8c525ca59d908afe4da157a4ad4826ad"
server_key_iv: "1268ac36e2c3b2a4a6cb9e1260e8972b"
server_key_version: "encryptionkey@0"
I20260925 01:30:40.021628 6516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "5d9edb25d92d4ae69ed3470a53935700"
format_stamp: "Formatted at 2026-09-25 01:30:40 on dist-test-slave-h6hf"
server_key: "8c525ca59d908afe4da157a4ad4826ad"
server_key_iv: "1268ac36e2c3b2a4a6cb9e1260e8972b"
server_key_version: "encryptionkey@0"
I20260925 01:30:40.026790 6516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:30:40.030295 6531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:40.031641 6516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:30:40.031780 6516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "5d9edb25d92d4ae69ed3470a53935700"
format_stamp: "Formatted at 2026-09-25 01:30:40 on dist-test-slave-h6hf"
server_key: "8c525ca59d908afe4da157a4ad4826ad"
server_key_iv: "1268ac36e2c3b2a4a6cb9e1260e8972b"
server_key_version: "encryptionkey@0"
I20260925 01:30:40.031944 6516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:30:40.055042 6516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:30:40.059268 6516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:30:40.059582 6516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:30:40.060557 6516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:30:40.061954 6516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:30:40.062033 6516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:40.062179 6516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:30:40.062230 6516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:30:40.088259 6376 heartbeater.cc:503] Master 127.27.160.62:37315 was elected leader, sending a full tablet report...
I20260925 01:30:40.096006 6516 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:38327
I20260925 01:30:40.096050 6643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:38327 every 8 connection(s)
I20260925 01:30:40.097410 6516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:30:40.104998 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 6516
I20260925 01:30:40.105247 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:30:40.105772 28288 external_mini_cluster.cc:1428] Setting key a678768fb7baa0d4678b7d8e87620c87
Sep 25 01:30:40 dist-test-slave-h6hf krb5kdc[6142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299840, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:30:40.118129 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.100112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:48449 (local address 127.27.160.62:37315)
0925 01:30:40.100562 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.100566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.101698 (+ 1132us) server_negotiation.cc:407] Connection header received
0925 01:30:40.103690 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.103694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.103772 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.103875 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:40.105707 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.106672 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.108391 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.108969 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.109066 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.112399 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:40.112404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:40.112408 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:40.112438 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:40.115281 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.116001 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.116005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.116008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.116127 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.116558 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.116561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.116563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.116808 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:40.116922 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.117678 (+ 756us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.117844 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260925 01:30:40.119237 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.100693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37315 (local address 127.27.160.3:48449)
0925 01:30:40.101525 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:40.101579 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:40.103401 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:40.104097 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:40.104105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:40.104630 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:30:40.105527 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.105553 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.106837 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.106841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.108257 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.108269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.109195 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.109201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.109396 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.110076 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:30:40.110098 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:30:40.112212 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:30:40.115414 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:40.115425 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:40.115438 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:40.115888 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:40.116244 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:30:40.116248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:30:40.116251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:30:40.116448 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:30:40.116939 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:30:40.116944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:30:40.117337 (+ 393us) client_negotiation.cc:769] Sending connection context
0925 01:30:40.117581 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 01:30:40.118051 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":372,"threads_started":1}
I20260925 01:30:40.120849 6644 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37315
I20260925 01:30:40.121263 6644 heartbeater.cc:464] Registering TS with master...
I20260925 01:30:40.122082 6644 heartbeater.cc:511] Master 127.27.160.62:37315 requested a full tablet report, sending...
I20260925 01:30:40.123382 6187 ts_manager.cc:194] Registered new tserver with Master: 5d9edb25d92d4ae69ed3470a53935700 (127.27.160.3:38327)
I20260925 01:30:40.124198 6187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:48449
I20260925 01:30:40.134166 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:30:40.150799 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.138850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.27.160.62:37315)
0925 01:30:40.139009 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.139013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.139540 (+ 527us) server_negotiation.cc:407] Connection header received
0925 01:30:40.139680 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.139683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.139747 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.139833 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:40.141240 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.141966 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.143174 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.143600 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.143693 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.146047 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:40.146052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:40.146055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:40.146085 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:40.148457 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.149167 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.149178 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.149182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.149308 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.149629 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.149632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.149634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.149933 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:40.150105 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.150406 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.150575 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:30:40.154851 6187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:30:40.162070 6187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:30:40.198865 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.174585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.27.160.1:42657)
0925 01:30:40.175764 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.175777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.187680 (+ 11903us) server_negotiation.cc:407] Connection header received
0925 01:30:40.188148 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.188156 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.188441 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.188720 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:40.190194 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.192377 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.196134 (+ 3757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.197703 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.197849 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.198106 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.198374 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.198544 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":959,"threads_started":1}
I20260925 01:30:40.198957 6659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.174441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:42657 (local address 127.0.0.1:47586)
0925 01:30:40.187517 (+ 13076us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:40.187565 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:40.187857 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:40.189089 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:40.189101 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:40.189179 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:40.189934 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.189951 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.192647 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.192656 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.195972 (+ 3316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.195986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.197900 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.197907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.198092 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.198149 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 01:30:40.198278 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 01:30:40.198464 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12770,"mutex_wait_us":93,"spinlock_wait_cycles":17280,"thread_start_us":12784,"threads_started":1}
I20260925 01:30:40.199982 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.175259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.27.160.3:38327)
0925 01:30:40.187048 (+ 11789us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.187061 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.187679 (+ 618us) server_negotiation.cc:407] Connection header received
0925 01:30:40.188179 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.188187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.188437 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.188652 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:40.190194 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.193655 (+ 3461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.197100 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.198554 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.198684 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.198900 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.199467 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.199644 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11576,"thread_start_us":11407,"threads_started":1}
I20260925 01:30:40.200853 6657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.175062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:38327 (local address 127.0.0.1:57566)
0925 01:30:40.187509 (+ 12447us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:40.187565 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:40.187857 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:40.189088 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:40.189100 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:40.189170 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:40.189885 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.189907 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.193886 (+ 3979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.193895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.196905 (+ 3010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.196924 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.199058 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.199065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.199263 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.199292 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:30:40.200241 (+ 949us) client_negotiation.cc:240] Negotiation successful
0925 01:30:40.200386 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12328,"mutex_wait_us":537,"spinlock_wait_cycles":7552,"thread_start_us":11982,"threads_started":1}
I20260925 01:30:40.203977 6661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.188316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41007 (local address 127.0.0.1:33456)
0925 01:30:40.196001 (+ 7685us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:40.196027 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:40.196146 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:40.197028 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:40.197033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:40.197059 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:40.197503 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.197516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.199379 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.199387 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.201344 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.201359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.203060 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.203067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.203231 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.203263 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:30:40.203368 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:30:40.203494 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7548,"thread_start_us":7400,"threads_started":1}
I20260925 01:30:40.203732 6311 tablet_service.cc:1467] Processing CreateTablet for tablet 166580a73e0c454794988369015e6642 (DEFAULT_TABLE table=test-table [id=b30fbe25faa54542b2cd5a18165f3dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:40.204550 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.188363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.27.160.2:41007)
0925 01:30:40.195656 (+ 7293us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.195668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.196193 (+ 525us) server_negotiation.cc:407] Connection header received
0925 01:30:40.196299 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.196307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.196601 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.196800 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:40.197697 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.199230 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.201491 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.202846 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.203003 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.203187 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.203434 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.203606 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7122,"thread_start_us":6985,"threads_started":1}
I20260925 01:30:40.204842 6579 tablet_service.cc:1467] Processing CreateTablet for tablet 166580a73e0c454794988369015e6642 (DEFAULT_TABLE table=test-table [id=b30fbe25faa54542b2cd5a18165f3dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:40.205531 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166580a73e0c454794988369015e6642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:40.206318 6579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166580a73e0c454794988369015e6642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:40.207578 6445 tablet_service.cc:1467] Processing CreateTablet for tablet 166580a73e0c454794988369015e6642 (DEFAULT_TABLE table=test-table [id=b30fbe25faa54542b2cd5a18165f3dcb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:30:40.208602 6445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 166580a73e0c454794988369015e6642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:30:40.222059 6664 tablet_bootstrap.cc:492] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Bootstrap starting.
I20260925 01:30:40.225660 6663 tablet_bootstrap.cc:492] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Bootstrap starting.
I20260925 01:30:40.226738 6665 tablet_bootstrap.cc:492] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Bootstrap starting.
I20260925 01:30:40.227525 6664 tablet_bootstrap.cc:654] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:40.228565 6664 log.cc:824] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:40.230156 6665 tablet_bootstrap.cc:654] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:40.231328 6664 tablet_bootstrap.cc:492] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: No bootstrap required, opened a new log
I20260925 01:30:40.231555 6665 log.cc:824] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:40.231624 6664 ts_tablet_manager.cc:1397] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:30:40.232651 6663 tablet_bootstrap.cc:654] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Neither blocks nor log segments found. Creating new log.
I20260925 01:30:40.234009 6663 log.cc:824] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:30:40.234853 6665 tablet_bootstrap.cc:492] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: No bootstrap required, opened a new log
I20260925 01:30:40.235110 6665 ts_tablet_manager.cc:1397] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:30:40.237237 6663 tablet_bootstrap.cc:492] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: No bootstrap required, opened a new log
I20260925 01:30:40.237542 6663 ts_tablet_manager.cc:1397] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260925 01:30:40.240096 6664 raft_consensus.cc:348] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.240460 6664 raft_consensus.cc:374] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:40.240598 6664 raft_consensus.cc:729] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be55fa2375d741e4b0370eeb94c4426b, State: Initialized, Role: FOLLOWER
I20260925 01:30:40.241257 6664 consensus_queue.cc:260] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.242990 6664 ts_tablet_manager.cc:1428] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 01:30:40.246274 6665 raft_consensus.cc:348] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.246731 6665 raft_consensus.cc:374] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:40.246835 6665 raft_consensus.cc:729] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9edb25d92d4ae69ed3470a53935700, State: Initialized, Role: FOLLOWER
I20260925 01:30:40.247442 6665 consensus_queue.cc:260] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.247926 6663 raft_consensus.cc:348] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.248445 6663 raft_consensus.cc:374] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:30:40.248549 6663 raft_consensus.cc:729] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a54b1706e01464e9f9e303ae2a7ceb3, State: Initialized, Role: FOLLOWER
I20260925 01:30:40.248637 6644 heartbeater.cc:503] Master 127.27.160.62:37315 was elected leader, sending a full tablet report...
I20260925 01:30:40.249398 6665 ts_tablet_manager.cc:1428] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 01:30:40.249194 6663 consensus_queue.cc:260] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.250365 6510 heartbeater.cc:503] Master 127.27.160.62:37315 was elected leader, sending a full tablet report...
I20260925 01:30:40.250910 6663 ts_tablet_manager.cc:1428] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
W20260925 01:30:40.306651 6377 tablet.cc:2369] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:30:40.329178 6511 tablet.cc:2369] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:30:40.349367 6645 tablet.cc:2369] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:30:40.581522 6670 raft_consensus.cc:482] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:30:40.581847 6670 raft_consensus.cc:504] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.583598 6670 leader_election.cc:288] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be55fa2375d741e4b0370eeb94c4426b (127.27.160.1:42657), 1a54b1706e01464e9f9e303ae2a7ceb3 (127.27.160.2:41007)
I20260925 01:30:40.590525 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.583930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:42657 (local address 127.27.160.3:59777)
0925 01:30:40.584133 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:40.584155 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:40.584288 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:40.585062 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:40.585067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:40.585102 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:40.585363 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.585377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.586686 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.586693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.588553 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.588569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.589958 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.589963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.590100 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.590125 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:30:40.590199 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 01:30:40.590298 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":25}
I20260925 01:30:40.590646 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.584137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:59777 (local address 127.27.160.1:42657)
0925 01:30:40.584339 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.584346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.584369 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:30:40.584444 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.584451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.584547 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.584638 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:40.585552 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.586549 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.588724 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.589811 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.589978 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.590055 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.590255 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.590390 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:30:40.591691 6672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.584333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41007 (local address 127.27.160.3:41265)
0925 01:30:40.585176 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 01:30:40.585195 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:30:40.585320 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:30:40.585669 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:30:40.585674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:30:40.585702 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:30:40.586050 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.586064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.587378 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.587383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.589204 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:30:40.589216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.590937 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.590945 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:30:40.591128 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.591158 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 01:30:40.591309 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 01:30:40.591421 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":1,"thread_start_us":535,"threads_started":1}
I20260925 01:30:40.591782 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.584745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:41265 (local address 127.27.160.2:41007)
0925 01:30:40.584980 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.584986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.585280 (+ 294us) server_negotiation.cc:407] Connection header received
0925 01:30:40.585449 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.585454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.585531 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.585624 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:30:40.586224 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.587220 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.589357 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.590595 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.590754 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.590826 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.591395 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.591542 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:30:40.591910 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166580a73e0c454794988369015e6642" candidate_uuid: "5d9edb25d92d4ae69ed3470a53935700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be55fa2375d741e4b0370eeb94c4426b" is_pre_election: true
I20260925 01:30:40.592581 6331 raft_consensus.cc:2454] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9edb25d92d4ae69ed3470a53935700 in term 0.
I20260925 01:30:40.593003 6465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166580a73e0c454794988369015e6642" candidate_uuid: "5d9edb25d92d4ae69ed3470a53935700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" is_pre_election: true
I20260925 01:30:40.593389 6534 leader_election.cc:302] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9edb25d92d4ae69ed3470a53935700, be55fa2375d741e4b0370eeb94c4426b; no voters:
I20260925 01:30:40.593622 6465 raft_consensus.cc:2454] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d9edb25d92d4ae69ed3470a53935700 in term 0.
I20260925 01:30:40.593916 6670 raft_consensus.cc:2781] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:30:40.594087 6670 raft_consensus.cc:482] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:30:40.594208 6670 raft_consensus.cc:3037] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:40.596424 6670 raft_consensus.cc:504] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.596997 6670 leader_election.cc:288] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [CANDIDATE]: Term 1 election: Requested vote from peers be55fa2375d741e4b0370eeb94c4426b (127.27.160.1:42657), 1a54b1706e01464e9f9e303ae2a7ceb3 (127.27.160.2:41007)
I20260925 01:30:40.597630 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166580a73e0c454794988369015e6642" candidate_uuid: "5d9edb25d92d4ae69ed3470a53935700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be55fa2375d741e4b0370eeb94c4426b"
I20260925 01:30:40.597657 6465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "166580a73e0c454794988369015e6642" candidate_uuid: "5d9edb25d92d4ae69ed3470a53935700" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3"
I20260925 01:30:40.597867 6331 raft_consensus.cc:3037] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:40.597858 6465 raft_consensus.cc:3037] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:30:40.601063 6331 raft_consensus.cc:2454] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9edb25d92d4ae69ed3470a53935700 in term 1.
I20260925 01:30:40.601063 6465 raft_consensus.cc:2454] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d9edb25d92d4ae69ed3470a53935700 in term 1.
I20260925 01:30:40.601480 6534 leader_election.cc:302] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9edb25d92d4ae69ed3470a53935700, be55fa2375d741e4b0370eeb94c4426b; no voters:
I20260925 01:30:40.601799 6670 raft_consensus.cc:2781] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:30:40.602337 6670 raft_consensus.cc:686] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 1 LEADER]: Becoming Leader. State: Replica: 5d9edb25d92d4ae69ed3470a53935700, State: Running, Role: LEADER
I20260925 01:30:40.602919 6670 consensus_queue.cc:237] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } }
I20260925 01:30:40.607916 6186 catalog_manager.cc:5495] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d9edb25d92d4ae69ed3470a53935700 (127.27.160.3). New cstate: current_term: 1 leader_uuid: "5d9edb25d92d4ae69ed3470a53935700" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d9edb25d92d4ae69ed3470a53935700" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 38327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 } health_report { overall_health: UNKNOWN } } }
I20260925 01:30:40.641805 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.636477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.27.160.3:38327)
0925 01:30:40.636687 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.636695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.636716 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:30:40.636819 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.636823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.636911 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.637046 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:30:40.637444 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.638184 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.639514 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.639980 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.640086 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.640588 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:30:40.641264 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:30:40.641399 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.641529 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.641601 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:30:40.650775 6331 raft_consensus.cc:1261] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 1 FOLLOWER]: Refusing update from remote peer 5d9edb25d92d4ae69ed3470a53935700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:40.650806 6465 raft_consensus.cc:1261] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 1 FOLLOWER]: Refusing update from remote peer 5d9edb25d92d4ae69ed3470a53935700: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:30:40.651923 6670 consensus_queue.cc:1035] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be55fa2375d741e4b0370eeb94c4426b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:40.652273 6673 consensus_queue.cc:1035] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a54b1706e01464e9f9e303ae2a7ceb3" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:30:40.679421 6186 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:40.679733 6186 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:30:40.683321 6186 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ed944fed3f84d01af8c34e4c76946ad: Sending DeleteTablet for 3 replicas of tablet 166580a73e0c454794988369015e6642
I20260925 01:30:40.684617 6579 tablet_service.cc:1514] Processing DeleteTablet for tablet 166580a73e0c454794988369015e6642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:40 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:57566
I20260925 01:30:40.684476 6311 tablet_service.cc:1514] Processing DeleteTablet for tablet 166580a73e0c454794988369015e6642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:40 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:47586
I20260925 01:30:40.685995 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 6248
I20260925 01:30:40.686991 6445 tablet_service.cc:1514] Processing DeleteTablet for tablet 166580a73e0c454794988369015e6642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:30:40 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:33456
I20260925 01:30:40.695940 6685 tablet_replica.cc:330] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: stopping tablet replica
I20260925 01:30:40.696633 6685 raft_consensus.cc:2229] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:30:40.697429 6685 raft_consensus.cc:2258] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:40.698330 6688 tablet_replica.cc:330] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: stopping tablet replica
I20260925 01:30:40.698834 6688 raft_consensus.cc:2229] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:40.699689 6688 raft_consensus.cc:2258] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:40.700670 6685 ts_tablet_manager.cc:1905] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:40.702046 6688 ts_tablet_manager.cc:1905] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:30:40.709777 6688 ts_tablet_manager.cc:1918] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:40.710078 6688 log.cc:1196] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/166580a73e0c454794988369015e6642
I20260925 01:30:40.710709 6688 ts_tablet_manager.cc:1939] T 166580a73e0c454794988369015e6642 P 1a54b1706e01464e9f9e303ae2a7ceb3: Deleting consensus metadata
I20260925 01:30:40.711719 6685 ts_tablet_manager.cc:1918] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:40.711916 6685 log.cc:1196] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/166580a73e0c454794988369015e6642
I20260925 01:30:40.712616 6685 ts_tablet_manager.cc:1939] T 166580a73e0c454794988369015e6642 P 5d9edb25d92d4ae69ed3470a53935700: Deleting consensus metadata
I20260925 01:30:40.713024 6173 catalog_manager.cc:4841] TS 1a54b1706e01464e9f9e303ae2a7ceb3 (127.27.160.2:41007): tablet 166580a73e0c454794988369015e6642 (table test-table [id=b30fbe25faa54542b2cd5a18165f3dcb]) successfully deleted
I20260925 01:30:40.713649 6686 tablet_replica.cc:330] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: stopping tablet replica
I20260925 01:30:40.714139 6686 raft_consensus.cc:2229] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:30:40.714720 6686 raft_consensus.cc:2258] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:30:40.716614 6172 catalog_manager.cc:4841] TS 5d9edb25d92d4ae69ed3470a53935700 (127.27.160.3:38327): tablet 166580a73e0c454794988369015e6642 (table test-table [id=b30fbe25faa54542b2cd5a18165f3dcb]) successfully deleted
I20260925 01:30:40.717684 6686 ts_tablet_manager.cc:1905] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 01:30:40 dist-test-slave-h6hf krb5kdc[6142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:30:40.725454 6686 ts_tablet_manager.cc:1918] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:30:40.725785 6658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.687478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43322 (local address 127.27.160.1:42657)
0925 01:30:40.687694 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.687704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.703886 (+ 16182us) server_negotiation.cc:407] Connection header received
0925 01:30:40.704981 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.704987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.705077 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.705179 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:40.708760 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.709936 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.713513 (+ 3577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.714146 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.714287 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.719535 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:40.719553 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:40.719574 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:40.719635 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:40.722758 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.723483 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.723491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.723494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.723638 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.724049 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.724053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.724055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.724410 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:40.724730 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.725043 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.725310 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:30:40.725783 6686 log.cc:1196] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/166580a73e0c454794988369015e6642
I20260925 01:30:40.726433 6686 ts_tablet_manager.cc:1939] T 166580a73e0c454794988369015e6642 P be55fa2375d741e4b0370eeb94c4426b: Deleting consensus metadata
I20260925 01:30:40.726545 6371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:40.817775 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 6248
I20260925 01:30:40.817965 6173 catalog_manager.cc:4841] TS be55fa2375d741e4b0370eeb94c4426b (127.27.160.1:42657): tablet 166580a73e0c454794988369015e6642 (table test-table [id=b30fbe25faa54542b2cd5a18165f3dcb]) successfully deleted
I20260925 01:30:40.836365 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 6382
Sep 25 01:30:40 dist-test-slave-h6hf krb5kdc[6142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:30:40.854404 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.837496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.27.160.2:41007)
0925 01:30:40.837705 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.837712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.837734 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:30:40.837804 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.837809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.837905 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.838013 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:40.839250 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.840347 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.841646 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.842480 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.842640 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.847021 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:40.847044 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:40.847068 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:40.847117 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:40.851292 (+ 4175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.852005 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.852017 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.852021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.852213 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.852593 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.852600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.852605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.853175 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:40.853495 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.853712 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.853957 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:30:40.855094 6505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:40.946244 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 6382
I20260925 01:30:40.963407 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 6516
Sep 25 01:30:40 dist-test-slave-h6hf krb5kdc[6142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:30:40.982982 6660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:30:40.964251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.27.160.3:38327)
0925 01:30:40.964431 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:30:40.964436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:30:40.964515 (+ 79us) server_negotiation.cc:407] Connection header received
0925 01:30:40.964721 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:30:40.964725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:30:40.964791 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:30:40.964878 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:30:40.966916 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.967803 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.969852 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:30:40.970268 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:30:40.970362 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:30:40.976368 (+ 6006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:30:40.976382 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:30:40.976398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:30:40.976447 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:30:40.979960 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.980682 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.980689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.980695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.980867 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:30:40.981337 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:30:40.981344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:30:40.981346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:30:40.981704 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:30:40.982014 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 01:30:40.982342 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 01:30:40.982591 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:30:40.983829 6639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:41.105546 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 6516
I20260925 01:30:41.124934 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 6155
I20260925 01:30:41.126269 6217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:30:41.236989 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 6155
2026-09-25T01:30:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4700 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:30:41 dist-test-slave-h6hf krb5kdc[6695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:30:41 dist-test-slave-h6hf krb5kdc[6695](info): set up 2 sockets
Sep 25 01:30:41 dist-test-slave-h6hf krb5kdc[6695](info): commencing operation
krb5kdc: starting...
W20260925 01:30:42.355069 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:30:42 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:30:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/rangeradmin_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/rangeradmin_127.27.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/rangerlookup_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/rangerlookup_127.27.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/HTTP_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/HTTP_127.27.160.20@KRBTEST.COM.keytab.
I20260925 01:30:42.708977 28288 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskI60hYW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/postgres -l logfile start
2026-09-25 01:30:45.394 UTC [6725] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 01:30:45.394 UTC [6725] LOG: listening on IPv4 address "127.27.160.20", port 33385
2026-09-25 01:30:45.398 UTC [6725] LOG: listening on Unix socket "/tmp/.s.PGSQL.33385"
2026-09-25 01:30:45.419 UTC [6728] LOG: database system was shut down at 2026-09-25 01:30:43 UTC
2026-09-25 01:30:45.423 UTC [6725] LOG: database system is ready to accept connections
I20260925 01:30:46.398665 28288 mini_postgres.cc:96] Postgres bound to 33385
2026-09-25 01:30:46.418 UTC [6736] FATAL: database "slave" does not exist
127.27.160.20:33385 - accepting connections
I20260925 01:30:46.419229 28288 mini_ranger.cc:161] Starting Ranger...
I20260925 01:30:46.448830 28288 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 01:30:46.971071 28288 mini_ranger.cc:87] Created ranger Postgres database
I20260925 01:30:46.971185 28288 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-admin
2026-09-25 01:30:47,296 [I] DB FLAVOR :POSTGRES
2026-09-25 01:30:47,297 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:30:47,297 [I] Checking connection..
2026-09-25 01:30:47,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 01:30:47,572 [I] Checking connection passed.
2026-09-25 01:30:47,573 [I] --------- Verifying version history table ---------
2026-09-25 01:30:47,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:30:47,829 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:30:47,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:30:48,075 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:30:48,076 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 01:30:48,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 01:30:48.324 UTC [6800] WARNING: there is no transaction in progress
2026-09-25 01:30:48,330 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 01:30:48,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:30:48,589 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 01:30:48,590 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:30:48,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 01:30:48,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 01:30:49,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h6hf.c.gcp-upstream.internal','N') ;"
2026-09-25 01:30:49,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:30:49,603 [I] Table x_portal_user does not exist in database ranger
2026-09-25 01:30:49,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:30:49,849 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 01:30:49,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:30:50,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:30:50,356 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 01:30:50,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 01:30:51.345 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.362 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.379 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.395 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.411 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.429 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.485 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.493 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.507 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.517 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.528 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.544 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.556 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.564 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.572 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:51.579 UTC [6908] WARNING: there is no transaction in progress
2026-09-25 01:30:52,264 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 01:30:52,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 01:30:52,522 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 01:30:52,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:30:52,786 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 01:30:52,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:30:53,056 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 01:30:53,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:30:53,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:30:53,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h6hf.c.gcp-upstream.internal';"
2026-09-25 01:30:53,826 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 01:30:53,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 01:30:54,104 [I] DB_PATCHES have already been applied
I20260925 01:30:54.110100 28288 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-admin:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskI60hYW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 01:30:54.110195 28288 mini_ranger.cc:193] Using host: 127.27.160.20
I20260925 01:30:54.126371 28288 mini_ranger.cc:227] Ranger admin URL: http://127.27.160.20:33185
Sep 25, 2026 1:30:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:30:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:30:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:30:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:30:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:30:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:30:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 01:30:55.289727 28288 mini_ranger.cc:160] Time spent starting Ranger: real 8.871s user 0.003s sys 0.042s
Sep 25, 2026 1:31:15 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:31:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 1:31:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:31:15 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:31:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 01:31:16.877626 28288 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/rangerkms_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/rangerkms_127.27.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.160.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.160.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.160.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/HTTP_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.160.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/HTTP_127.27.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 01:31:17 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 01:31:17.080689 28288 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 01:31:17.103484 28288 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 01:31:18.051875 28288 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 01:31:18.052045 28288 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms
I20260925 01:31:18.052188 28288 mini_ranger_kms.cc:227] Using postgres at 127.27.160.20:33385
2026-09-25 01:31:19,072 [I] DB FLAVOR :POSTGRES
2026-09-25 01:31:19,072 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:31:19,073 [I] Checking connection
2026-09-25 01:31:19,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 01:31:19,324 [I] connection success
2026-09-25 01:31:19,324 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 01:31:19,324 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 01:31:19,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 01:31:19,589 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 01:31:19,589 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:31:19,589 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 01:31:19,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:33385/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 01:31:19,862 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 01:31:20.360862 28288 mini_ranger_kms.cc:310] Created kms service
I20260925 01:31:20.585678 28288 mini_ranger_kms.cc:326] Created kudu user
I20260925 01:31:20.632720 28288 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 01:31:20 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:31:21.108282 28288 mini_ranger_kms.cc:384] Added ranger policy
I20260925 01:31:21.108623 28288 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms:/tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 01:31:21.108680 28288 mini_ranger_kms.cc:240] Using host: 127.27.160.20
I20260925 01:31:21.132670 28288 mini_ranger_kms.cc:276] Ranger KMS PID: 7089
I20260925 01:31:21.132899 28288 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.160.20:55489
Sep 25, 2026 1:31:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:31:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:31:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:31:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:31:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:31:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 01:31:23 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.160.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:31:23.511798 28288 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.431s user 0.151s sys 0.428s
I20260925 01:31:23.512086 28288 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 01:31:23.512323 28288 mini_ranger_kms.cc:401] 127.27.160.20:55489/kms/v1/keys
Sep 25 01:31:23 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.160.20@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
Sep 25 01:31:24 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299883, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.160.20@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
Sep 25 01:31:26 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:31:26.742136 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:35195
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:35195
--ranger_config_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:55489/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:31:27.084411 7139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:31:27.084764 7139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:31:27.084928 7139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:31:27.092157 7139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:31:27.092319 7139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:31:27.092428 7139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:31:27.092538 7139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:31:27.092625 7139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:31:27.102859 7139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.160.20:55489/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:35195
--ranger_config_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:35195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:31:27.104247 7139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:31:27.105644 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:31:27.117816 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:31:27.117862 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:31:27.118001 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:31:27.118817 7139 server_base.cc:1034] running on GCE node
I20260925 01:31:27.119601 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:31:27.121060 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:31:27.122313 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790299887122283 us; error 31 us; skew 500 ppm
I20260925 01:31:27.122668 7139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:31:27 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299887, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:31:27.131078 7139 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:31:27.133292 7139 webserver.cc:460] Webserver started at http://127.27.160.62:44769/ using document root <none> and password file <none>
I20260925 01:31:27.134284 7139 fs_manager.cc:359] Metadata directory not provided
I20260925 01:31:27.134426 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:31:27.134825 7139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:31:27 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299887, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:31:27.189628 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "96589d5cb7c54c74898e3addd7cc32ba"
format_stamp: "Formatted at 2026-09-25 01:31:27 on dist-test-slave-h6hf"
server_key: "bcf800f9fe210ddde6fc65340b07f960"
server_key_iv: "163fed7cdec3e9d5bd66f2f99ec0f037"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:27.190616 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "96589d5cb7c54c74898e3addd7cc32ba"
format_stamp: "Formatted at 2026-09-25 01:31:27 on dist-test-slave-h6hf"
server_key: "bcf800f9fe210ddde6fc65340b07f960"
server_key_iv: "163fed7cdec3e9d5bd66f2f99ec0f037"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:27.196756 7139 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260925 01:31:27.213431 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:27.215065 7139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:31:27.215271 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "96589d5cb7c54c74898e3addd7cc32ba"
format_stamp: "Formatted at 2026-09-25 01:31:27 on dist-test-slave-h6hf"
server_key: "bcf800f9fe210ddde6fc65340b07f960"
server_key_iv: "163fed7cdec3e9d5bd66f2f99ec0f037"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:27.215533 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:31:27.230827 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:31:27.234969 7139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:31:27.235288 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:31:27.261072 7139 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:35195
I20260925 01:31:27.261091 7205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:35195 every 8 connection(s)
I20260925 01:31:27.262575 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:31:27.263797 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 7139
I20260925 01:31:27.264020 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:31:27.264847 28288 external_mini_cluster.cc:1428] Setting key 96d22ad3d40b27f7ccd64f1e212dd34a
I20260925 01:31:27.268594 7206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:31:27.286656 7206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba: Bootstrap starting.
I20260925 01:31:27.290444 7206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba: Neither blocks nor log segments found. Creating new log.
I20260925 01:31:27.291651 7206 log.cc:824] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:31:27 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:31:27.295313 7206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba: No bootstrap required, opened a new log
I20260925 01:31:27.304877 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:27.267095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.27.160.62:35195)
0925 01:31:27.279444 (+ 12349us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:27.279462 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:27.279520 (+ 58us) server_negotiation.cc:407] Connection header received
0925 01:31:27.282033 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:27.282067 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:27.282708 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:27.283242 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:31:27.284990 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:27.287037 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:27.288897 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:27.290006 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:27.290210 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:27.295312 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:31:27.295324 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:31:27.295345 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:31:27.295393 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:31:27.299688 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:27.300695 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:27.300705 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:27.300715 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:27.300937 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:27.301427 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:27.301432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:27.301436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:27.302205 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:31:27.302507 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:27.303006 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 01:31:27.303443 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12060,"thread_start_us":11860,"threads_started":1}
I20260925 01:31:27.304725 7206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96589d5cb7c54c74898e3addd7cc32ba" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35195 } }
I20260925 01:31:27.305163 7206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:31:27.305281 7206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96589d5cb7c54c74898e3addd7cc32ba, State: Initialized, Role: FOLLOWER
I20260925 01:31:27.306030 7206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96589d5cb7c54c74898e3addd7cc32ba" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35195 } }
I20260925 01:31:27.306289 7206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:31:27.306488 7206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:31:27.306684 7206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:31:27.309396 7206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96589d5cb7c54c74898e3addd7cc32ba" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35195 } }
I20260925 01:31:27.309960 7206 leader_election.cc:302] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96589d5cb7c54c74898e3addd7cc32ba; no voters:
I20260925 01:31:27.310675 7206 leader_election.cc:288] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:31:27.310786 7211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:31:27.312180 7211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [term 1 LEADER]: Becoming Leader. State: Replica: 96589d5cb7c54c74898e3addd7cc32ba, State: Running, Role: LEADER
I20260925 01:31:27.312806 7211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96589d5cb7c54c74898e3addd7cc32ba" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35195 } }
I20260925 01:31:27.313025 7206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:31:27.322959 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96589d5cb7c54c74898e3addd7cc32ba. Latest consensus state: current_term: 1 leader_uuid: "96589d5cb7c54c74898e3addd7cc32ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96589d5cb7c54c74898e3addd7cc32ba" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35195 } } }
I20260925 01:31:27.323374 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [sys.catalog]: This master's current role is: LEADER
I20260925 01:31:27.324018 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96589d5cb7c54c74898e3addd7cc32ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96589d5cb7c54c74898e3addd7cc32ba" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35195 } } }
I20260925 01:31:27.324307 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba [sys.catalog]: This master's current role is: LEADER
I20260925 01:31:27.324508 7218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:31:27.346659 7218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:31:27.363035 7206 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 01:31:27.363767 7218 catalog_manager.cc:1347] Generated new cluster ID: 345dac6296f844e8ab4a87b03d88423e
I20260925 01:31:27.363880 7218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:31:27.392827 7218 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:31:27.394975 7218 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:31:27.407001 7218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba: Generated new TSK 0
I20260925 01:31:27.408265 7218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:31:28.209458 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:55489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35195
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:31:28 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 01:31:28.696393 7251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:31:28.696830 7251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:31:28.697064 7251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:31:28.708738 7251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:31:28.708926 7251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:31:28.709146 7251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:31:28.725198 7251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.160.20:55489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:31:28.727042 7251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:31:28.728883 7251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:31:28.745003 7262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:31:28.745028 7264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:31:28.746335 7261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:31:28.747120 7251 server_base.cc:1034] running on GCE node
I20260925 01:31:28.747893 7251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:31:28.750622 7251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:31:28.751981 7251 hybrid_clock.cc:648] HybridClock initialized: now 1790299888751896 us; error 64 us; skew 500 ppm
I20260925 01:31:28.752358 7251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:31:28 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:31:28.755739 7251 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:31:28.757737 7251 webserver.cc:460] Webserver started at http://127.27.160.1:41197/ using document root <none> and password file <none>
I20260925 01:31:28.758657 7251 fs_manager.cc:359] Metadata directory not provided
I20260925 01:31:28.758875 7251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:31:28.759315 7251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:31:28 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:31:28.779443 7251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "aee38b1422c741a2ac7cc1de78abb5c8"
format_stamp: "Formatted at 2026-09-25 01:31:28 on dist-test-slave-h6hf"
server_key: "3855fb01f0991622394e31b715ae65b1"
server_key_iv: "5b37291671946d301aa507e0b09f95ef"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:28.780761 7251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "aee38b1422c741a2ac7cc1de78abb5c8"
format_stamp: "Formatted at 2026-09-25 01:31:28 on dist-test-slave-h6hf"
server_key: "3855fb01f0991622394e31b715ae65b1"
server_key_iv: "5b37291671946d301aa507e0b09f95ef"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:28.793768 7251 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.008s sys 0.003s
I20260925 01:31:28.827459 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:28.829363 7251 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.009s sys 0.008s
I20260925 01:31:28.829586 7251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "aee38b1422c741a2ac7cc1de78abb5c8"
format_stamp: "Formatted at 2026-09-25 01:31:28 on dist-test-slave-h6hf"
server_key: "3855fb01f0991622394e31b715ae65b1"
server_key_iv: "5b37291671946d301aa507e0b09f95ef"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:28.829829 7251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:31:28.871881 7251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:31:28.876596 7251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:31:28.876924 7251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:31:28.878093 7251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:31:28.879567 7251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:31:28.879665 7251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:28.879809 7251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:31:28.879876 7251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:28.943068 7251 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:45325
I20260925 01:31:28.943115 7383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:45325 every 8 connection(s)
I20260925 01:31:28.944833 7251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:31:28.950369 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 7251
I20260925 01:31:28.950623 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:31:28.951452 28288 external_mini_cluster.cc:1428] Setting key 127fd12bdab33c0813641b9d3f844f9b
Sep 25 01:31:29 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:31:29.019301 7387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:28.949699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:35053 (local address 127.27.160.62:35195)
0925 01:31:28.980611 (+ 30912us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:28.980619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:28.993182 (+ 12563us) server_negotiation.cc:407] Connection header received
0925 01:31:28.997722 (+ 4540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:28.997729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:28.997829 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:28.997910 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:31:29.000859 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:29.002045 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:29.004548 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:29.005219 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:29.005320 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:29.011063 (+ 5743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:31:29.011072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:31:29.011077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:31:29.011121 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:31:29.014945 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:29.016090 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:29.016097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:29.016101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:29.016263 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:29.016916 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:29.016922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:29.016926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:29.017314 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:31:29.017466 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:29.018565 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0925 01:31:29.018821 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30756,"thread_start_us":30927,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
I20260925 01:31:29.022064 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:28.948727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:35195 (local address 127.27.160.1:35053)
0925 01:31:28.992183 (+ 43456us) negotiation.cc:107] Waiting for socket to connect
0925 01:31:28.992247 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 01:31:28.996286 (+ 4039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:31:28.998503 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:31:28.998516 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:31:28.999200 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:31:29.000062 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:29.000087 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:29.002270 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:29.002275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:29.004355 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:29.004372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:29.005708 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:29.005716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:29.006029 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:29.007254 (+ 1225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:31:29.007295 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:31:29.010708 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:31:29.015218 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:31:29.015228 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:31:29.015249 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:31:29.015917 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:31:29.016435 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:31:29.016440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:31:29.016457 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:31:29.016771 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:31:29.017495 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:31:29.017504 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:31:29.018072 (+ 568us) client_negotiation.cc:769] Sending connection context
0925 01:31:29.019642 (+ 1570us) client_negotiation.cc:240] Negotiation successful
0925 01:31:29.020353 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43047,"thread_start_us":47831,"threads_started":1}
I20260925 01:31:29.024513 7384 heartbeater.cc:347] Connected to a master server at 127.27.160.62:35195
I20260925 01:31:29.024848 7384 heartbeater.cc:464] Registering TS with master...
I20260925 01:31:29.025739 7384 heartbeater.cc:511] Master 127.27.160.62:35195 requested a full tablet report, sending...
I20260925 01:31:29.028473 7171 ts_manager.cc:194] Registered new tserver with Master: aee38b1422c741a2ac7cc1de78abb5c8 (127.27.160.1:45325)
I20260925 01:31:29.030558 7171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:35053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:31:29.060870 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:55489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35195
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:31:29 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
Sep 25 01:31:29 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299888, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
W20260925 01:31:29.524670 7390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:31:29.525216 7390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:31:29.525435 7390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:31:29.538421 7390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:31:29.538697 7390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:31:29.538996 7390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:31:29.552292 7390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.160.20:55489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:31:29.554337 7390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:31:29.556198 7390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:31:29.581548 7397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:31:29.582708 7398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:31:29.592257 7400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:31:29.592483 7390 server_base.cc:1034] running on GCE node
I20260925 01:31:29.593600 7390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:31:29.598197 7390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:31:29.599575 7390 hybrid_clock.cc:648] HybridClock initialized: now 1790299889599498 us; error 79 us; skew 500 ppm
I20260925 01:31:29.600085 7390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:31:29 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299889, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:31:29.604609 7390 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:31:29.613759 7390 webserver.cc:460] Webserver started at http://127.27.160.2:36107/ using document root <none> and password file <none>
I20260925 01:31:29.614789 7390 fs_manager.cc:359] Metadata directory not provided
I20260925 01:31:29.614960 7390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:31:29.615361 7390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:31:29 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299889, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:31:29.641623 7390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "9574c02a9c8f442eb20c0a63de48129c"
format_stamp: "Formatted at 2026-09-25 01:31:29 on dist-test-slave-h6hf"
server_key: "8e342618ee809d16d379cd9ef4a66844"
server_key_iv: "80b13e67ef0101d0930814177e0fa6ad"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:29.642781 7390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "9574c02a9c8f442eb20c0a63de48129c"
format_stamp: "Formatted at 2026-09-25 01:31:29 on dist-test-slave-h6hf"
server_key: "8e342618ee809d16d379cd9ef4a66844"
server_key_iv: "80b13e67ef0101d0930814177e0fa6ad"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:29.650635 7390 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260925 01:31:29.688174 7407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:29.690332 7390 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.003s
I20260925 01:31:29.690752 7390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "9574c02a9c8f442eb20c0a63de48129c"
format_stamp: "Formatted at 2026-09-25 01:31:29 on dist-test-slave-h6hf"
server_key: "8e342618ee809d16d379cd9ef4a66844"
server_key_iv: "80b13e67ef0101d0930814177e0fa6ad"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:29.691288 7390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:31:29.715116 7390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:31:29.719460 7390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:31:29.719771 7390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:31:29.720984 7390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:31:29.722596 7390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:31:29.722697 7390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:29.722847 7390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:31:29.722919 7390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:29.758315 7390 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:36945
I20260925 01:31:29.758384 7526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:36945 every 8 connection(s)
I20260925 01:31:29.759600 7390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:31:29.762401 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 7390
I20260925 01:31:29.762586 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:31:29.763111 28288 external_mini_cluster.cc:1428] Setting key a41e0c32c4aab73cf953e7b4de8c426e
Sep 25 01:31:29 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299889, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:31:29.787323 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:29.762495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:39643 (local address 127.27.160.62:35195)
0925 01:31:29.768070 (+ 5575us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:29.768074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:29.768093 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:31:29.768153 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:29.768157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:29.768217 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:29.768356 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:31:29.770400 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:29.771634 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:29.773955 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:29.774711 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:29.774860 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:29.779219 (+ 4359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:31:29.779229 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:31:29.779237 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:31:29.779285 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:31:29.783132 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:29.784197 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:29.784206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:29.784212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:29.784386 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:29.784960 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:29.784968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:29.784974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:29.785461 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:31:29.785620 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:29.786564 (+ 944us) server_negotiation.cc:299] Negotiation successful
0925 01:31:29.786820 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5473,"thread_start_us":5373,"threads_started":1}
I20260925 01:31:29.788446 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:29.763116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:35195 (local address 127.27.160.2:39643)
0925 01:31:29.765768 (+ 2652us) negotiation.cc:107] Waiting for socket to connect
0925 01:31:29.765814 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:31:29.767498 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:31:29.768667 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:31:29.768680 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:31:29.769310 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:31:29.770180 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:29.770210 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:29.771814 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:29.771819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:29.773775 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:29.773795 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:29.774947 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:29.774954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:29.775243 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:29.776075 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:31:29.776104 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:31:29.778917 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:31:29.783318 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:31:29.783328 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:31:29.783349 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:31:29.784009 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:31:29.784518 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:31:29.784523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:31:29.784527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:31:29.784822 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:31:29.785628 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:31:29.785637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:31:29.786101 (+ 464us) client_negotiation.cc:769] Sending connection context
0925 01:31:29.786432 (+ 331us) client_negotiation.cc:240] Negotiation successful
0925 01:31:29.787006 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2354,"thread_start_us":2131,"threads_started":1}
I20260925 01:31:29.790647 7527 heartbeater.cc:347] Connected to a master server at 127.27.160.62:35195
I20260925 01:31:29.790995 7527 heartbeater.cc:464] Registering TS with master...
I20260925 01:31:29.791913 7527 heartbeater.cc:511] Master 127.27.160.62:35195 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:31:29.793735 7171 ts_manager.cc:194] Registered new tserver with Master: 9574c02a9c8f442eb20c0a63de48129c (127.27.160.2:36945)
I20260925 01:31:29.794556 7171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:39643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:31:29.825496 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:55489/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35195
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:31:30.034806 7384 heartbeater.cc:503] Master 127.27.160.62:35195 was elected leader, sending a full tablet report...
W20260925 01:31:30.166402 7533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:31:30.166706 7533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:31:30.166877 7533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:31:30.174281 7533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:31:30.174459 7533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:31:30.174640 7533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:31:30.184708 7533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.160.20:55489/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:31:30.186115 7533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:31:30.187446 7533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:31:30.201165 7538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:31:30.201220 7539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:31:30.201242 7541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:31:30.202279 7533 server_base.cc:1034] running on GCE node
I20260925 01:31:30.203001 7533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:31:30.204064 7533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:31:30.205261 7533 hybrid_clock.cc:648] HybridClock initialized: now 1790299890205177 us; error 69 us; skew 500 ppm
I20260925 01:31:30.205624 7533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:31:30 dist-test-slave-h6hf krb5kdc[6695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299890, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:31:30.209018 7533 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:31:30.210590 7533 webserver.cc:460] Webserver started at http://127.27.160.3:33299/ using document root <none> and password file <none>
I20260925 01:31:30.211319 7533 fs_manager.cc:359] Metadata directory not provided
I20260925 01:31:30.211469 7533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:31:30.211781 7533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:31:30 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299890, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:31:30.229564 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "b6daf8821b1545ababb53b2c14ffe0ff"
format_stamp: "Formatted at 2026-09-25 01:31:30 on dist-test-slave-h6hf"
server_key: "0338b70bc710695cf317b93f51655264"
server_key_iv: "efe942fbdd52f0c41a89bd9367b32fbd"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:30.230440 7533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "b6daf8821b1545ababb53b2c14ffe0ff"
format_stamp: "Formatted at 2026-09-25 01:31:30 on dist-test-slave-h6hf"
server_key: "0338b70bc710695cf317b93f51655264"
server_key_iv: "efe942fbdd52f0c41a89bd9367b32fbd"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:30.236008 7533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:31:30.253612 7548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:30.255546 7533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260925 01:31:30.255786 7533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "b6daf8821b1545ababb53b2c14ffe0ff"
format_stamp: "Formatted at 2026-09-25 01:31:30 on dist-test-slave-h6hf"
server_key: "0338b70bc710695cf317b93f51655264"
server_key_iv: "efe942fbdd52f0c41a89bd9367b32fbd"
server_key_version: "kuduclusterkey@0"
I20260925 01:31:30.256057 7533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:31:30.303750 7533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:31:30.307765 7533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:31:30.308035 7533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:31:30.308916 7533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:31:30.310295 7533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:31:30.310370 7533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:30.310454 7533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:31:30.310508 7533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:31:30.344173 7533 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:36677
I20260925 01:31:30.344224 7660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:36677 every 8 connection(s)
I20260925 01:31:30.345574 7533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:31:30.350629 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 7533
I20260925 01:31:30.350827 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:31:30.351264 28288 external_mini_cluster.cc:1428] Setting key 29129d21ed3a4376d93d93157b4f784e
Sep 25 01:31:30 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299890, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:31:30.367530 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.348480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:47229 (local address 127.27.160.62:35195)
0925 01:31:30.348965 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.348969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.350015 (+ 1046us) server_negotiation.cc:407] Connection header received
0925 01:31:30.351989 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.351995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.352096 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.352212 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:31:30.354092 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.355238 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.356863 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.357484 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.357591 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.360903 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:31:30.360909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:31:30.360912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:31:30.360969 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:31:30.364055 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:30.364835 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:30.364841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:30.364845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:30.364981 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:30.365448 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:30.365454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:30.365460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:30.365789 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:31:30.365964 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.366885 (+ 921us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.367166 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":233,"threads_started":1}
I20260925 01:31:30.368577 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.349041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:35195 (local address 127.27.160.3:47229)
0925 01:31:30.349854 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 01:31:30.349902 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:31:30.351667 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:31:30.352496 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:31:30.352509 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:31:30.353186 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:31:30.353907 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.353926 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.355387 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.355391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.356708 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.356720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.357688 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.357693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.357899 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.358540 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:31:30.358561 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:31:30.360677 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:31:30.364227 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:31:30.364237 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:31:30.364260 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:31:30.364713 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:31:30.365097 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:31:30.365102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:31:30.365106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:31:30.365340 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:31:30.365939 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:31:30.365947 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:31:30.366406 (+ 459us) client_negotiation.cc:769] Sending connection context
0925 01:31:30.366761 (+ 355us) client_negotiation.cc:240] Negotiation successful
0925 01:31:30.367302 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":280,"threads_started":1}
I20260925 01:31:30.370253 7661 heartbeater.cc:347] Connected to a master server at 127.27.160.62:35195
I20260925 01:31:30.370640 7661 heartbeater.cc:464] Registering TS with master...
I20260925 01:31:30.371523 7661 heartbeater.cc:511] Master 127.27.160.62:35195 requested a full tablet report, sending...
I20260925 01:31:30.373200 7171 ts_manager.cc:194] Registered new tserver with Master: b6daf8821b1545ababb53b2c14ffe0ff (127.27.160.3:36677)
I20260925 01:31:30.374063 7171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:47229
I20260925 01:31:30.379974 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:31:30.399223 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.384844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.27.160.62:35195)
0925 01:31:30.385074 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.385084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.385588 (+ 504us) server_negotiation.cc:407] Connection header received
0925 01:31:30.385714 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.385718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.385805 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.385908 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:31:30.387464 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.388311 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.389882 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.390306 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.390431 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.393031 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:31:30.393038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:31:30.393042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:31:30.393095 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:31:30.396089 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:30.396982 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:30.396987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:30.396990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:30.397218 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:30.397685 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:30.397689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:30.397692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:30.398096 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:31:30.398296 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.398660 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.398846 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:31:30.403964 7171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:31:30.412263 7171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:31:30.450013 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.430778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.27.160.1:45325)
0925 01:31:30.431385 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.431396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.442202 (+ 10806us) server_negotiation.cc:407] Connection header received
0925 01:31:30.442364 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.442373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.442583 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.443417 (+ 834us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:31:30.443709 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.445298 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.447863 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.448805 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.448964 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.449259 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.449482 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.449651 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":270,"threads_started":1}
I20260925 01:31:30.450089 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.430172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:45325 (local address 127.0.0.1:55044)
0925 01:31:30.442040 (+ 11868us) negotiation.cc:107] Waiting for socket to connect
0925 01:31:30.442064 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:31:30.442214 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:31:30.442896 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:31:30.442907 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:31:30.442958 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:31:30.443550 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.443561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.445469 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.445474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.447719 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.447730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.449017 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.449023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.449234 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.449291 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 01:31:30.449435 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 01:31:30.449634 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11685,"mutex_wait_us":32,"thread_start_us":11015,"threads_started":1}
I20260925 01:31:30.451565 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.429885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.27.160.3:36677)
0925 01:31:30.430889 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.430896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.441489 (+ 10593us) server_negotiation.cc:407] Connection header received
0925 01:31:30.441981 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.441990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.442295 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.443659 (+ 1364us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:31:30.444620 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.446158 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.448728 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.450178 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.450291 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.450540 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.451091 (+ 551us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.451240 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":687,"threads_started":1}
I20260925 01:31:30.451566 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.429796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:36677 (local address 127.0.0.1:33778)
0925 01:31:30.441219 (+ 11423us) negotiation.cc:107] Waiting for socket to connect
0925 01:31:30.441287 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 01:31:30.441606 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:31:30.444207 (+ 2601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:31:30.444210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:31:30.444229 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:31:30.444459 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.444466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.446324 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.446329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.448564 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.448577 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.450745 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.450754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.450920 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.450943 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:31:30.451041 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 01:31:30.451163 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10988,"mutex_wait_us":4,"thread_start_us":11888,"threads_started":1}
I20260925 01:31:30.452939 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.442477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:36945 (local address 127.0.0.1:39166)
0925 01:31:30.443060 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 01:31:30.443079 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:31:30.443203 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:31:30.444824 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:31:30.444829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:31:30.444855 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:31:30.445252 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.445264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.447089 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.447094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.449602 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.449617 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.452140 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.452146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.452311 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.452342 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:31:30.452437 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 01:31:30.452574 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"spinlock_wait_cycles":7552,"thread_start_us":317,"threads_started":1}
I20260925 01:31:30.452975 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.442514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.27.160.2:36945)
0925 01:31:30.444281 (+ 1767us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.444290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.444316 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:31:30.444402 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.444420 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.444634 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.444825 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:31:30.445457 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.446931 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.450303 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.451676 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.451809 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.451999 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.452503 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.452668 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1608,"thread_start_us":1502,"threads_started":1}
I20260925 01:31:30.455066 7319 tablet_service.cc:1467] Processing CreateTablet for tablet 0409a28ed5c141e89d9f50726c4b607c (DEFAULT_TABLE table=test-table [id=178f06eeb16a4aea80de272b3e0f43f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:31:30.455732 7596 tablet_service.cc:1467] Processing CreateTablet for tablet 0409a28ed5c141e89d9f50726c4b607c (DEFAULT_TABLE table=test-table [id=178f06eeb16a4aea80de272b3e0f43f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:31:30.456605 7319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0409a28ed5c141e89d9f50726c4b607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:31:30.456629 7462 tablet_service.cc:1467] Processing CreateTablet for tablet 0409a28ed5c141e89d9f50726c4b607c (DEFAULT_TABLE table=test-table [id=178f06eeb16a4aea80de272b3e0f43f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:31:30.457381 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0409a28ed5c141e89d9f50726c4b607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:31:30.457991 7462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0409a28ed5c141e89d9f50726c4b607c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:31:30.468822 7680 tablet_bootstrap.cc:492] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Bootstrap starting.
I20260925 01:31:30.472589 7681 tablet_bootstrap.cc:492] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Bootstrap starting.
I20260925 01:31:30.473018 7680 tablet_bootstrap.cc:654] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Neither blocks nor log segments found. Creating new log.
I20260925 01:31:30.474555 7680 log.cc:824] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:31:30.474622 7682 tablet_bootstrap.cc:492] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Bootstrap starting.
I20260925 01:31:30.478581 7682 tablet_bootstrap.cc:654] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Neither blocks nor log segments found. Creating new log.
I20260925 01:31:30.478868 7680 tablet_bootstrap.cc:492] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: No bootstrap required, opened a new log
I20260925 01:31:30.479099 7681 tablet_bootstrap.cc:654] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Neither blocks nor log segments found. Creating new log.
I20260925 01:31:30.479183 7680 ts_tablet_manager.cc:1397] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260925 01:31:30.479959 7682 log.cc:824] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:31:30.480341 7681 log.cc:824] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Log is configured to *not* fsync() on all Append() calls
I20260925 01:31:30.483419 7682 tablet_bootstrap.cc:492] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: No bootstrap required, opened a new log
I20260925 01:31:30.483712 7682 ts_tablet_manager.cc:1397] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 01:31:30.484787 7681 tablet_bootstrap.cc:492] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: No bootstrap required, opened a new log
I20260925 01:31:30.485071 7681 ts_tablet_manager.cc:1397] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260925 01:31:30.487296 7680 raft_consensus.cc:348] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.487696 7680 raft_consensus.cc:374] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:31:30.487833 7680 raft_consensus.cc:729] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee38b1422c741a2ac7cc1de78abb5c8, State: Initialized, Role: FOLLOWER
I20260925 01:31:30.488477 7680 consensus_queue.cc:260] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.490389 7680 ts_tablet_manager.cc:1428] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260925 01:31:30.495055 7682 raft_consensus.cc:348] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.495555 7682 raft_consensus.cc:374] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:31:30.495666 7682 raft_consensus.cc:729] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9574c02a9c8f442eb20c0a63de48129c, State: Initialized, Role: FOLLOWER
I20260925 01:31:30.495723 7681 raft_consensus.cc:348] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.496161 7681 raft_consensus.cc:374] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:31:30.496258 7681 raft_consensus.cc:729] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6daf8821b1545ababb53b2c14ffe0ff, State: Initialized, Role: FOLLOWER
I20260925 01:31:30.496521 7682 consensus_queue.cc:260] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.496820 7681 consensus_queue.cc:260] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.497768 7527 heartbeater.cc:503] Master 127.27.160.62:35195 was elected leader, sending a full tablet report...
I20260925 01:31:30.498039 7661 heartbeater.cc:503] Master 127.27.160.62:35195 was elected leader, sending a full tablet report...
I20260925 01:31:30.498663 7681 ts_tablet_manager.cc:1428] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260925 01:31:30.499840 7682 ts_tablet_manager.cc:1428] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260925 01:31:30.502173 7385 tablet.cc:2369] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:31:30.511658 7528 tablet.cc:2369] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:31:30.597411 7662 tablet.cc:2369] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:31:30.622480 7688 raft_consensus.cc:482] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:31:30.622757 7688 raft_consensus.cc:504] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.624410 7688 leader_election.cc:288] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee38b1422c741a2ac7cc1de78abb5c8 (127.27.160.1:45325), 9574c02a9c8f442eb20c0a63de48129c (127.27.160.2:36945)
I20260925 01:31:30.630888 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.624903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:43225 (local address 127.27.160.1:45325)
0925 01:31:30.625216 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.625222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.625242 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:31:30.625357 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.625364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.625455 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.625556 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:31:30.626039 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.627079 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.628900 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.629985 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.630118 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.630171 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.630504 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.630599 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:31:30.630910 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.624663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:45325 (local address 127.27.160.3:43225)
0925 01:31:30.624897 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 01:31:30.624915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:31:30.625074 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:31:30.625649 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:31:30.625659 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:31:30.625685 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:31:30.625883 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.625897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.627210 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.627215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.628746 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.628757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.630135 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.630142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.630326 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.630361 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:31:30.630474 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:31:30.630618 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 01:31:30.631703 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.625399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:43557 (local address 127.27.160.2:36945)
0925 01:31:30.625588 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.625596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.625862 (+ 266us) server_negotiation.cc:407] Connection header received
0925 01:31:30.626025 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.626030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.626094 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.626162 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:31:30.626740 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.627611 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.629636 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.630606 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.630746 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.630819 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.631323 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.631451 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:31:30.631735 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.625072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:36945 (local address 127.27.160.3:43557)
0925 01:31:30.625753 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 01:31:30.625766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:31:30.625872 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:31:30.626208 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:31:30.626212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:31:30.626235 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:31:30.626600 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.626611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.627773 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.627777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.629483 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:31:30.629496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.630878 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.630887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:31:30.631092 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.631129 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:31:30.631288 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 01:31:30.631420 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":314,"threads_started":1}
I20260925 01:31:30.632066 7339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0409a28ed5c141e89d9f50726c4b607c" candidate_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" is_pre_election: true
I20260925 01:31:30.632666 7339 raft_consensus.cc:2454] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6daf8821b1545ababb53b2c14ffe0ff in term 0.
I20260925 01:31:30.632750 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0409a28ed5c141e89d9f50726c4b607c" candidate_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9574c02a9c8f442eb20c0a63de48129c" is_pre_election: true
I20260925 01:31:30.633338 7482 raft_consensus.cc:2454] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6daf8821b1545ababb53b2c14ffe0ff in term 0.
I20260925 01:31:30.633378 7551 leader_election.cc:302] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee38b1422c741a2ac7cc1de78abb5c8, b6daf8821b1545ababb53b2c14ffe0ff; no voters:
I20260925 01:31:30.633955 7688 raft_consensus.cc:2781] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:31:30.634109 7688 raft_consensus.cc:482] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:31:30.634245 7688 raft_consensus.cc:3037] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:31:30.636559 7688 raft_consensus.cc:504] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.637214 7688 leader_election.cc:288] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [CANDIDATE]: Term 1 election: Requested vote from peers aee38b1422c741a2ac7cc1de78abb5c8 (127.27.160.1:45325), 9574c02a9c8f442eb20c0a63de48129c (127.27.160.2:36945)
I20260925 01:31:30.637692 7339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0409a28ed5c141e89d9f50726c4b607c" candidate_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee38b1422c741a2ac7cc1de78abb5c8"
I20260925 01:31:30.637737 7482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0409a28ed5c141e89d9f50726c4b607c" candidate_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9574c02a9c8f442eb20c0a63de48129c"
I20260925 01:31:30.637905 7339 raft_consensus.cc:3037] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:31:30.637941 7482 raft_consensus.cc:3037] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:31:30.640102 7339 raft_consensus.cc:2454] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6daf8821b1545ababb53b2c14ffe0ff in term 1.
I20260925 01:31:30.640115 7482 raft_consensus.cc:2454] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6daf8821b1545ababb53b2c14ffe0ff in term 1.
I20260925 01:31:30.640527 7551 leader_election.cc:302] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9574c02a9c8f442eb20c0a63de48129c, b6daf8821b1545ababb53b2c14ffe0ff; no voters:
I20260925 01:31:30.640851 7688 raft_consensus.cc:2781] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:31:30.641341 7688 raft_consensus.cc:686] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 1 LEADER]: Becoming Leader. State: Replica: b6daf8821b1545ababb53b2c14ffe0ff, State: Running, Role: LEADER
I20260925 01:31:30.641814 7688 consensus_queue.cc:237] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } }
I20260925 01:31:30.646633 7170 catalog_manager.cc:5495] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff reported cstate change: term changed from 0 to 1, leader changed from <none> to b6daf8821b1545ababb53b2c14ffe0ff (127.27.160.3). New cstate: current_term: 1 leader_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6daf8821b1545ababb53b2c14ffe0ff" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 } health_report { overall_health: UNKNOWN } } }
I20260925 01:31:30.695245 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.689012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.27.160.3:36677)
0925 01:31:30.689245 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.689253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.689331 (+ 78us) server_negotiation.cc:407] Connection header received
0925 01:31:30.689620 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.689624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.689713 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.689878 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:31:30.690313 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.691117 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.692845 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.693373 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.693529 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.694089 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:31:30.694624 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:31:30.694796 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.694858 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.694954 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:31:30.704227 7482 raft_consensus.cc:1261] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 1 FOLLOWER]: Refusing update from remote peer b6daf8821b1545ababb53b2c14ffe0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:31:30.704231 7339 raft_consensus.cc:1261] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 1 FOLLOWER]: Refusing update from remote peer b6daf8821b1545ababb53b2c14ffe0ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:31:30.705338 7688 consensus_queue.cc:1035] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "9574c02a9c8f442eb20c0a63de48129c" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:31:30.705751 7690 consensus_queue.cc:1035] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee38b1422c741a2ac7cc1de78abb5c8" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:31:30.718839 7693 mvcc.cc:205] Tried to move back new op lower bound from 7333068352312639488 to 7333068352072617984. Current Snapshot: MvccSnapshot[applied={T|T < 7333068352312639488}]
I20260925 01:31:30.721392 7694 mvcc.cc:205] Tried to move back new op lower bound from 7333068352312639488 to 7333068352072617984. Current Snapshot: MvccSnapshot[applied={T|T < 7333068352312639488}]
I20260925 01:31:30.735807 7170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:31:30.736088 7170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:31:30.739547 7170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96589d5cb7c54c74898e3addd7cc32ba: Sending DeleteTablet for 3 replicas of tablet 0409a28ed5c141e89d9f50726c4b607c
I20260925 01:31:30.740705 7596 tablet_service.cc:1514] Processing DeleteTablet for tablet 0409a28ed5c141e89d9f50726c4b607c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:31:30 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:33778
I20260925 01:31:30.740870 7462 tablet_service.cc:1514] Processing DeleteTablet for tablet 0409a28ed5c141e89d9f50726c4b607c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:31:30 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:39166
I20260925 01:31:30.741750 7319 tablet_service.cc:1514] Processing DeleteTablet for tablet 0409a28ed5c141e89d9f50726c4b607c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:31:30 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:55044
I20260925 01:31:30.742796 7705 tablet_replica.cc:330] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: stopping tablet replica
I20260925 01:31:30.743353 7705 raft_consensus.cc:2229] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:31:30.743940 7705 raft_consensus.cc:2258] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:31:30.743973 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 7251
I20260925 01:31:30.746657 7705 ts_tablet_manager.cc:1905] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:31:30.753430 7704 tablet_replica.cc:330] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: stopping tablet replica
I20260925 01:31:30.754057 7704 raft_consensus.cc:2229] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:31:30.754721 7704 raft_consensus.cc:2258] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:31:30.755740 7705 ts_tablet_manager.cc:1918] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:31:30.755936 7705 log.cc:1196] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/0409a28ed5c141e89d9f50726c4b607c
I20260925 01:31:30.756608 7705 ts_tablet_manager.cc:1939] T 0409a28ed5c141e89d9f50726c4b607c P aee38b1422c741a2ac7cc1de78abb5c8: Deleting consensus metadata
I20260925 01:31:30.757800 7704 ts_tablet_manager.cc:1905] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:31:30.758746 7157 catalog_manager.cc:4841] TS aee38b1422c741a2ac7cc1de78abb5c8 (127.27.160.1:45325): tablet 0409a28ed5c141e89d9f50726c4b607c (table test-table [id=178f06eeb16a4aea80de272b3e0f43f0]) successfully deleted
I20260925 01:31:30.768335 7704 ts_tablet_manager.cc:1918] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:31:30.768591 7704 log.cc:1196] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/0409a28ed5c141e89d9f50726c4b607c
I20260925 01:31:30.769281 7704 ts_tablet_manager.cc:1939] T 0409a28ed5c141e89d9f50726c4b607c P 9574c02a9c8f442eb20c0a63de48129c: Deleting consensus metadata
I20260925 01:31:30.771479 7157 catalog_manager.cc:4841] TS 9574c02a9c8f442eb20c0a63de48129c (127.27.160.2:36945): tablet 0409a28ed5c141e89d9f50726c4b607c (table test-table [id=178f06eeb16a4aea80de272b3e0f43f0]) successfully deleted
I20260925 01:31:30.772065 7703 tablet_replica.cc:330] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: stopping tablet replica
I20260925 01:31:30.772634 7703 raft_consensus.cc:2229] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:31:30.774026 7703 raft_consensus.cc:2258] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:31:30.776644 7703 ts_tablet_manager.cc:1905] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 01:31:30 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:31:30.785508 7703 ts_tablet_manager.cc:1918] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:31:30.785722 7703 log.cc:1196] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/0409a28ed5c141e89d9f50726c4b607c
I20260925 01:31:30.786290 7703 ts_tablet_manager.cc:1939] T 0409a28ed5c141e89d9f50726c4b607c P b6daf8821b1545ababb53b2c14ffe0ff: Deleting consensus metadata
I20260925 01:31:30.788533 7158 catalog_manager.cc:4841] TS b6daf8821b1545ababb53b2c14ffe0ff (127.27.160.3:36677): tablet 0409a28ed5c141e89d9f50726c4b607c (table test-table [id=178f06eeb16a4aea80de272b3e0f43f0]) successfully deleted
I20260925 01:31:30.793583 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.746027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.27.160.1:45325)
0925 01:31:30.746217 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.746225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.770789 (+ 24564us) server_negotiation.cc:407] Connection header received
0925 01:31:30.771820 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.771828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.771961 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.772061 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:31:30.774271 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.775451 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.777643 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.778408 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.778524 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.785206 (+ 6682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:31:30.785234 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:31:30.785257 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:31:30.785320 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:31:30.789622 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:30.790793 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:30.790804 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:30.790808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:30.790995 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:30.791573 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:30.791582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:30.791589 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:30.792130 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:31:30.792392 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.792872 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.793071 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:31:30.794273 7379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:31:30.888013 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 7251
I20260925 01:31:30.905937 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 7390
Sep 25 01:31:30 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:31:30.922051 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:30.906750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.27.160.2:36945)
0925 01:31:30.906920 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:30.906927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:30.907005 (+ 78us) server_negotiation.cc:407] Connection header received
0925 01:31:30.907246 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:30.907251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:30.907315 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:30.907445 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:31:30.908697 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.909501 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.910846 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:30.911238 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:30.911368 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:30.916067 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:31:30.916082 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:31:30.916099 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:31:30.916139 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:31:30.919187 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:30.919889 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:30.919897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:30.919900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:30.920030 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:30.920469 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:30.920475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:30.920479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:30.920859 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:31:30.921168 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:30.921480 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 01:31:30.921697 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:31:30.922672 7522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:31:31.014569 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 7390
I20260925 01:31:31.037778 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 7533
Sep 25 01:31:31 dist-test-slave-h6hf krb5kdc[6695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:31:31.056372 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:31:31.038719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.27.160.3:36677)
0925 01:31:31.038895 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:31:31.038905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:31:31.038968 (+ 63us) server_negotiation.cc:407] Connection header received
0925 01:31:31.039032 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:31:31.039037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:31:31.039147 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:31:31.039260 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:31:31.040676 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:31.041560 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:31.042946 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:31:31.043398 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:31:31.043553 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:31:31.049778 (+ 6225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:31:31.049795 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:31:31.049816 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:31:31.049866 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:31:31.053062 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:31.053962 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:31.053971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:31.053973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:31.054157 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:31:31.054573 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:31:31.054578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:31:31.054581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:31:31.054970 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:31:31.055330 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0925 01:31:31.055632 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 01:31:31.055868 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:31:31.057308 7656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:31:31.181252 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 7533
I20260925 01:31:31.200610 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 7139
I20260925 01:31:31.201808 7201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:31:31.315591 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 7139
I20260925 01:31:31.337353 28288 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 01:31:31.673237 28288 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 01:31:31.673518 28288 mini_ranger.cc:66] Stopping Ranger...
I20260925 01:31:32.019385 28288 mini_ranger.cc:68] Stopped Ranger
2026-09-25 01:31:32.019 UTC [6725] LOG: received smart shutdown request
2026-09-25 01:31:32.023 UTC [6725] LOG: background worker "logical replication launcher" (PID 6734) exited with exit code 1
2026-09-25 01:31:32.023 UTC [6729] LOG: shutting down
2026-09-25 01:31:32.201 UTC [6725] LOG: database system is shut down
2026-09-25T01:31:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51090 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:31:32 dist-test-slave-h6hf krb5kdc[7719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:31:32 dist-test-slave-h6hf krb5kdc[7719](info): set up 2 sockets
Sep 25 01:31:32 dist-test-slave-h6hf krb5kdc[7719](info): commencing operation
krb5kdc: starting...
W20260925 01:31:33.454574 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:31:33 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:31:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:31:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/rangeradmin_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/rangeradmin_127.27.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/rangerlookup_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/rangerlookup_127.27.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/HTTP_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/HTTP_127.27.160.20@KRBTEST.COM.keytab.
I20260925 01:31:33.807629 28288 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskI60hYW/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/postgres -l logfile start
2026-09-25 01:31:36.493 UTC [7749] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 01:31:36.493 UTC [7749] LOG: listening on IPv4 address "127.27.160.20", port 37125
2026-09-25 01:31:36.497 UTC [7749] LOG: listening on Unix socket "/tmp/.s.PGSQL.37125"
2026-09-25 01:31:36.523 UTC [7752] LOG: database system was shut down at 2026-09-25 01:31:34 UTC
2026-09-25 01:31:36.528 UTC [7749] LOG: database system is ready to accept connections
I20260925 01:31:37.610181 28288 mini_postgres.cc:96] Postgres bound to 37125
2026-09-25 01:31:37.628 UTC [7760] FATAL: database "slave" does not exist
127.27.160.20:37125 - accepting connections
I20260925 01:31:37.629340 28288 mini_ranger.cc:161] Starting Ranger...
I20260925 01:31:37.658283 28288 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 01:31:38.220204 28288 mini_ranger.cc:87] Created ranger Postgres database
I20260925 01:31:38.220329 28288 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-admin
2026-09-25 01:31:38,534 [I] DB FLAVOR :POSTGRES
2026-09-25 01:31:38,534 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:31:38,534 [I] Checking connection..
2026-09-25 01:31:38,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 01:31:38,806 [I] Checking connection passed.
2026-09-25 01:31:38,807 [I] --------- Verifying version history table ---------
2026-09-25 01:31:38,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:31:39,058 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:31:39,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:31:39,323 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:31:39,323 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 01:31:39,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 01:31:39.585 UTC [7824] WARNING: there is no transaction in progress
2026-09-25 01:31:39,593 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 01:31:39,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:31:39,837 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 01:31:39,838 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:31:39,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 01:31:40,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 01:31:40,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h6hf.c.gcp-upstream.internal','N') ;"
2026-09-25 01:31:40,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:31:40,867 [I] Table x_portal_user does not exist in database ranger
2026-09-25 01:31:40,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:31:41,124 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 01:31:41,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:31:41,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:31:41,651 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 01:31:41,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 01:31:42.742 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.760 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.778 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.796 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.815 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.838 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.903 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.911 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.927 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.937 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.949 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.965 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.976 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.983 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:42.991 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:43.000 UTC [7932] WARNING: there is no transaction in progress
2026-09-25 01:31:43,715 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 01:31:43,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 01:31:43,958 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 01:31:43,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:31:44,214 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 01:31:44,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:31:44,469 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 01:31:44,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:31:44,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:31:45,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h6hf.c.gcp-upstream.internal';"
2026-09-25 01:31:45,263 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 01:31:45,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 01:31:45,531 [I] DB_PATCHES have already been applied
I20260925 01:31:45.539556 28288 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-admin:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskI60hYW/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 01:31:45.539714 28288 mini_ranger.cc:193] Using host: 127.27.160.20
I20260925 01:31:45.556092 28288 mini_ranger.cc:227] Ranger admin URL: http://127.27.160.20:59629
Sep 25, 2026 1:31:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:31:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:31:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:31:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:31:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:31:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:31:46 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 01:31:46.728726 28288 mini_ranger.cc:160] Time spent starting Ranger: real 9.099s user 0.002s sys 0.045s
Sep 25, 2026 1:32:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:32:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 1:32:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:32:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:32:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 01:32:08.697921 28288 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.160.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.160.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/rangerkms_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.160.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/rangerkms_127.27.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.160.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.160.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.160.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/HTTP_127.27.160.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.160.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/HTTP_127.27.160.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 01:32:08 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299928, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 01:32:08.882687 28288 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 01:32:08.906771 28288 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 01:32:09.874097 28288 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 01:32:09.874233 28288 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms
I20260925 01:32:09.874302 28288 mini_ranger_kms.cc:227] Using postgres at 127.27.160.20:37125
2026-09-25 01:32:10,236 [I] DB FLAVOR :POSTGRES
2026-09-25 01:32:10,237 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:32:10,237 [I] Checking connection
2026-09-25 01:32:10,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 01:32:10,505 [I] connection success
2026-09-25 01:32:10,505 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 01:32:10,505 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 01:32:10,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 01:32:10,770 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 01:32:10,771 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:32:10,771 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 01:32:10,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.160.20:37125/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 01:32:11,051 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 01:32:11.564605 28288 mini_ranger_kms.cc:310] Created kms service
I20260925 01:32:11.799154 28288 mini_ranger_kms.cc:326] Created kudu user
I20260925 01:32:11.850857 28288 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 01:32:11 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299928, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:12.268886 28288 mini_ranger_kms.cc:384] Added ranger policy
I20260925 01:32:12.269271 28288 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms:/tmp/dist-test-taskI60hYW/build/asan/bin/postgresql.jar:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskI60hYW/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 01:32:12.269341 28288 mini_ranger_kms.cc:240] Using host: 127.27.160.20
I20260925 01:32:12.292167 28288 mini_ranger_kms.cc:276] Ranger KMS PID: 8113
I20260925 01:32:12.292317 28288 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.160.20:36869
Sep 25, 2026 1:32:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:32:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:32:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:32:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:32:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:32:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 01:32:14.683391 28288 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.801s user 0.095s sys 0.285s
I20260925 01:32:14.683629 28288 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 01:32:14.683795 28288 mini_ranger_kms.cc:401] 127.27.160.20:36869/kms/v1/keys
Sep 25 01:32:14 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299934, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.160.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 01:32:15 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299934, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.160.20@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
Sep 25 01:32:15 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299934, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.160.20@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
Sep 25 01:32:17 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:32:18.010002 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40105
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:40105
--ranger_config_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:36869/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:18.357347 8163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:18.357745 8163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:18.357923 8163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:18.365806 8163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:18.365940 8163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:18.366032 8163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:18.366127 8163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:18.366247 8163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:18.377310 8163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.160.20:36869/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:40105
--ranger_config_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:18.378765 8163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:18.380290 8163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:18.393530 8171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:18.393767 8169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:18.393509 8168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:18.394578 8163 server_base.cc:1034] running on GCE node
I20260925 01:32:18.395368 8163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:18.396776 8163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:18.397976 8163 hybrid_clock.cc:648] HybridClock initialized: now 1790299938397929 us; error 39 us; skew 500 ppm
I20260925 01:32:18.398290 8163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:18 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299938, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:18.405288 8163 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:32:18.406796 8163 webserver.cc:460] Webserver started at http://127.27.160.62:37627/ using document root <none> and password file <none>
I20260925 01:32:18.407459 8163 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:18.407557 8163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:18.407836 8163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:32:18 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299938, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:18.472287 8163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "607d2734698f47b196d4921673d80dc0"
format_stamp: "Formatted at 2026-09-25 01:32:18 on dist-test-slave-h6hf"
server_key: "5cbae6a78f0ea3c6d5c478972cd6a7ab"
server_key_iv: "f40b7796a3217ea591bb981edb08afe0"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:18.473489 8163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "607d2734698f47b196d4921673d80dc0"
format_stamp: "Formatted at 2026-09-25 01:32:18 on dist-test-slave-h6hf"
server_key: "5cbae6a78f0ea3c6d5c478972cd6a7ab"
server_key_iv: "f40b7796a3217ea591bb981edb08afe0"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:18.480715 8163 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260925 01:32:18.501925 8178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:18.503798 8163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:32:18.504014 8163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "607d2734698f47b196d4921673d80dc0"
format_stamp: "Formatted at 2026-09-25 01:32:18 on dist-test-slave-h6hf"
server_key: "5cbae6a78f0ea3c6d5c478972cd6a7ab"
server_key_iv: "f40b7796a3217ea591bb981edb08afe0"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:18.504273 8163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:18.539702 8163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:18.545250 8163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:18.545547 8163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:18.565259 8163 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:40105
I20260925 01:32:18.565362 8229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:40105 every 8 connection(s)
I20260925 01:32:18.566826 8163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:32:18.570156 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 8163
I20260925 01:32:18.570399 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:32:18.571110 28288 external_mini_cluster.cc:1428] Setting key 7690cc8da52489ecffee52bd06fc8d81
I20260925 01:32:18.571918 8230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:18.589794 8230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Bootstrap starting.
Sep 25 01:32:18 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:18.594568 8230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:18.596217 8230 log.cc:824] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:18.600334 8230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: No bootstrap required, opened a new log
I20260925 01:32:18.601641 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:18.573605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44528 (local address 127.27.160.62:40105)
0925 01:32:18.574670 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:18.574689 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:18.574748 (+ 59us) server_negotiation.cc:407] Connection header received
0925 01:32:18.577727 (+ 2979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:18.577770 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:18.578490 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:18.579097 (+ 607us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:18.580564 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:18.583011 (+ 2447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:18.584685 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:18.586110 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:18.586361 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:18.591247 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:18.591259 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:18.591283 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:18.591350 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:18.596159 (+ 4809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:18.597421 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:18.597432 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:18.597439 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:18.597603 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:18.598162 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:18.598167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:18.598170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:18.598918 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:18.599217 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:18.599725 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 01:32:18.600159 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":505,"threads_started":1}
I20260925 01:32:18.609565 8230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } }
I20260925 01:32:18.610168 8230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:18.610280 8230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607d2734698f47b196d4921673d80dc0, State: Initialized, Role: FOLLOWER
I20260925 01:32:18.611109 8230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } }
I20260925 01:32:18.611421 8230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:32:18.611578 8230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:32:18.611829 8230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:18.614424 8230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } }
I20260925 01:32:18.614982 8230 leader_election.cc:302] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 607d2734698f47b196d4921673d80dc0; no voters:
I20260925 01:32:18.615568 8230 leader_election.cc:288] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:32:18.615758 8235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:32:18.616618 8235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 607d2734698f47b196d4921673d80dc0, State: Running, Role: LEADER
I20260925 01:32:18.617249 8235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } }
I20260925 01:32:18.617977 8230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:32:18.620359 8237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 607d2734698f47b196d4921673d80dc0. Latest consensus state: current_term: 1 leader_uuid: "607d2734698f47b196d4921673d80dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } } }
I20260925 01:32:18.620445 8236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "607d2734698f47b196d4921673d80dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } } }
I20260925 01:32:18.620836 8237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:18.620877 8236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:18.621905 8241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:32:18.629462 8241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:32:18.630426 8230 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 01:32:18.649943 8241 catalog_manager.cc:1347] Generated new cluster ID: 8b78e2ce5dbf43d2914912f7297a494e
I20260925 01:32:18.650152 8241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:32:18.672719 8241 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:32:18.675030 8241 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:32:18.693903 8241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Generated new TSK 0
I20260925 01:32:18.694968 8241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:32:19.389616 8272 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:32:19.486310 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:36869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40105
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:32:19 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 01:32:19.913899 8275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:19.914233 8275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:19.914397 8275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:19.923635 8275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:19.923878 8275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:19.924093 8275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:32:19.936794 8275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.160.20:36869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:19.938320 8275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:19.939693 8275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:19.959681 8286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:19.960690 8285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:19.969902 8275 server_base.cc:1034] running on GCE node
W20260925 01:32:19.970373 8288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:19.971235 8275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:19.972421 8275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:19.973680 8275 hybrid_clock.cc:648] HybridClock initialized: now 1790299939973610 us; error 66 us; skew 500 ppm
I20260925 01:32:19.974067 8275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:19 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:19.977632 8275 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:32:19.979918 8275 webserver.cc:460] Webserver started at http://127.27.160.1:44271/ using document root <none> and password file <none>
I20260925 01:32:19.980722 8275 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:19.980876 8275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:19.981272 8275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:32:19 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:20.008533 8275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "321fe3e175934380a3126cb1018c4aa7"
format_stamp: "Formatted at 2026-09-25 01:32:19 on dist-test-slave-h6hf"
server_key: "0a4793cf5f52559e2e76c736f5f7c0e2"
server_key_iv: "d479dcb70ff4b3cef149dc866ef85473"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:20.010555 8275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "321fe3e175934380a3126cb1018c4aa7"
format_stamp: "Formatted at 2026-09-25 01:32:19 on dist-test-slave-h6hf"
server_key: "0a4793cf5f52559e2e76c736f5f7c0e2"
server_key_iv: "d479dcb70ff4b3cef149dc866ef85473"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:20.031491 8275 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.007s sys 0.012s
I20260925 01:32:20.070941 8295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:20.073342 8275 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.006s sys 0.012s
I20260925 01:32:20.073664 8275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "321fe3e175934380a3126cb1018c4aa7"
format_stamp: "Formatted at 2026-09-25 01:32:19 on dist-test-slave-h6hf"
server_key: "0a4793cf5f52559e2e76c736f5f7c0e2"
server_key_iv: "d479dcb70ff4b3cef149dc866ef85473"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:20.073927 8275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:20.098239 8275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:20.103462 8275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:20.103878 8275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:20.105020 8275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:20.113238 8275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:20.113371 8275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 01:32:20.113619 8275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:20.113762 8275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:20.173691 8275 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:35071
I20260925 01:32:20.175355 8275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:32:20.173691 8407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:35071 every 8 connection(s)
I20260925 01:32:20.183362 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 8275
I20260925 01:32:20.183609 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:32:20.184352 28288 external_mini_cluster.cc:1428] Setting key 206db9e575787fb4045ced1cdfddeac8
Sep 25 01:32:20 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:20.218616 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:20.180841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40105 (local address 127.27.160.1:47623)
0925 01:32:20.193895 (+ 13054us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:20.193971 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:20.196651 (+ 2680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:20.197895 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:20.197904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:20.198400 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:20.199124 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:20.199153 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:20.200470 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:20.200480 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:20.202450 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:20.202480 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:20.203950 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:20.203960 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:20.204314 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:20.205209 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:20.205248 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:20.208213 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:20.212829 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:20.212842 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:20.212866 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:20.213631 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:20.214158 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:20.214166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:20.214172 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:20.214445 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:20.215270 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:20.215281 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:20.215847 (+ 566us) client_negotiation.cc:769] Sending connection context
0925 01:32:20.216186 (+ 339us) client_negotiation.cc:240] Negotiation successful
0925 01:32:20.216889 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12649,"thread_start_us":12386,"threads_started":1}
I20260925 01:32:20.220189 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:20.180338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:47623 (local address 127.27.160.62:40105)
0925 01:32:20.183938 (+ 3600us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:20.183946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:20.197234 (+ 13288us) server_negotiation.cc:407] Connection header received
0925 01:32:20.197322 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:20.197328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:20.197441 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:20.197591 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:20.199384 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:20.200300 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:20.202658 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:20.203428 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:20.203582 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:20.208549 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:20.208559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:20.208567 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:20.208629 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:20.212615 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:20.213814 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:20.213824 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:20.213830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:20.214006 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:20.214585 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:20.214593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:20.214599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:20.215024 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:20.215180 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:20.219407 (+ 4227us) server_negotiation.cc:299] Negotiation successful
0925 01:32:20.219693 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3411,"thread_start_us":4823,"threads_started":1}
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:32:20.223007 8408 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40105
I20260925 01:32:20.223385 8408 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:20.224329 8408 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
I20260925 01:32:20.227018 8195 ts_manager.cc:194] Registered new tserver with Master: 321fe3e175934380a3126cb1018c4aa7 (127.27.160.1:35071)
I20260925 01:32:20.228657 8195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:47623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:32:20.261482 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:36869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40105
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:32:20 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
Sep 25 01:32:20 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299939, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
W20260925 01:32:20.772145 8414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:20.772698 8414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:20.772891 8414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:20.784632 8414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:20.784891 8414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:20.785137 8414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:32:20.801550 8414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.160.20:36869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:20.803568 8414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:20.805465 8414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:20.830184 8421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:20.841616 8422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:20.851707 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:20.852784 8414 server_base.cc:1034] running on GCE node
I20260925 01:32:20.854003 8414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:20.861053 8414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:20.869355 8414 hybrid_clock.cc:648] HybridClock initialized: now 1790299940869216 us; error 123 us; skew 500 ppm
I20260925 01:32:20.870008 8414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:20 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299940, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:20.874912 8414 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:32:20.880085 8414 webserver.cc:460] Webserver started at http://127.27.160.2:35763/ using document root <none> and password file <none>
I20260925 01:32:20.881209 8414 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:20.881392 8414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:20.881786 8414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:32:20 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299940, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:20.903514 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "05bc40d2999e493c8938f2ba8d5423e6"
format_stamp: "Formatted at 2026-09-25 01:32:20 on dist-test-slave-h6hf"
server_key: "e620852dd519efc8a6abc40e042093d3"
server_key_iv: "364d7b046f375b80fe7319a6d18e36a8"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:20.904589 8414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "05bc40d2999e493c8938f2ba8d5423e6"
format_stamp: "Formatted at 2026-09-25 01:32:20 on dist-test-slave-h6hf"
server_key: "e620852dd519efc8a6abc40e042093d3"
server_key_iv: "364d7b046f375b80fe7319a6d18e36a8"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:20.914111 8414 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20260925 01:32:20.941841 8431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:20.944146 8414 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.006s
I20260925 01:32:20.944470 8414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "05bc40d2999e493c8938f2ba8d5423e6"
format_stamp: "Formatted at 2026-09-25 01:32:20 on dist-test-slave-h6hf"
server_key: "e620852dd519efc8a6abc40e042093d3"
server_key_iv: "364d7b046f375b80fe7319a6d18e36a8"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:20.944839 8414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:20.972373 8414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:20.978616 8414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:20.978995 8414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:20.983666 8414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:20.985752 8414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:20.985867 8414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:20.986020 8414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:20.986086 8414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:21.057228 8414 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:41937
I20260925 01:32:21.057258 8550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:41937 every 8 connection(s)
I20260925 01:32:21.058990 8414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:32:21.064945 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 8414
I20260925 01:32:21.065189 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:32:21.065659 28288 external_mini_cluster.cc:1428] Setting key cc0aaf07ff33c5e28c81ee242e0ab9f9
Sep 25 01:32:21 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299940, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:21.086673 8553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:21.061915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:43609 (local address 127.27.160.62:40105)
0925 01:32:21.062448 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:21.062455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:21.063494 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 01:32:21.066278 (+ 2784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:21.066284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:21.066386 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:21.066508 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:21.068972 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.070270 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.072940 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.073733 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.073892 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:21.078519 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:21.078525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:21.078530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:21.078571 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:21.082447 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.083626 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.083632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.083636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.083812 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.084411 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.084416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.084419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.084809 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:21.084957 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:21.086004 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0925 01:32:21.086254 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260925 01:32:21.088075 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:21.062456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40105 (local address 127.27.160.2:43609)
0925 01:32:21.063299 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:21.063352 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:21.065866 (+ 2514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:21.067170 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:21.067184 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:21.067840 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:21.068716 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:21.068744 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.070457 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.070462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:21.072718 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:21.072747 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.074008 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.074015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:21.074331 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:21.075228 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:21.075259 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:21.078169 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:21.082674 (+ 4505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:21.082688 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:21.082712 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:21.083470 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:21.083973 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:21.083981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:21.083987 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:21.084272 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:21.084992 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:21.085004 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:21.085550 (+ 546us) client_negotiation.cc:769] Sending connection context
0925 01:32:21.085884 (+ 334us) client_negotiation.cc:240] Negotiation successful
0925 01:32:21.086522 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":323,"threads_started":1}
I20260925 01:32:21.089869 8551 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40105
I20260925 01:32:21.090242 8551 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:21.091176 8551 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
I20260925 01:32:21.093226 8195 ts_manager.cc:194] Registered new tserver with Master: 05bc40d2999e493c8938f2ba8d5423e6 (127.27.160.2:41937)
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:32:21.094614 8195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:43609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:32:21.125070 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:36869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40105
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:32:21.239017 8408 heartbeater.cc:503] Master 127.27.160.62:40105 was elected leader, sending a full tablet report...
W20260925 01:32:21.473067 8557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:21.473582 8557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:21.473713 8557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:21.481901 8557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:21.482043 8557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:21.482161 8557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:32:21.492496 8557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.160.20:36869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:21.493933 8557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:21.495247 8557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:21.509413 8563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:21.509490 8565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:21.509383 8562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:21.510324 8557 server_base.cc:1034] running on GCE node
I20260925 01:32:21.511019 8557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:21.512060 8557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:21.513317 8557 hybrid_clock.cc:648] HybridClock initialized: now 1790299941513262 us; error 56 us; skew 500 ppm
I20260925 01:32:21.513684 8557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:21 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:21.517392 8557 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:32:21.519284 8557 webserver.cc:460] Webserver started at http://127.27.160.3:37181/ using document root <none> and password file <none>
I20260925 01:32:21.520052 8557 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:21.520210 8557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:21.520545 8557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:32:21 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:21.538763 8557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "82741e5dfc904ba9b5799c074034b2f4"
format_stamp: "Formatted at 2026-09-25 01:32:21 on dist-test-slave-h6hf"
server_key: "6062fbe7574954bd91bc230d3e5a1b76"
server_key_iv: "0428602710f4f9998695e4a9a78328c5"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:21.540071 8557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "82741e5dfc904ba9b5799c074034b2f4"
format_stamp: "Formatted at 2026-09-25 01:32:21 on dist-test-slave-h6hf"
server_key: "6062fbe7574954bd91bc230d3e5a1b76"
server_key_iv: "0428602710f4f9998695e4a9a78328c5"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:21.546924 8557 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 01:32:21.565964 8572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:21.568192 8557 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260925 01:32:21.568420 8557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "82741e5dfc904ba9b5799c074034b2f4"
format_stamp: "Formatted at 2026-09-25 01:32:21 on dist-test-slave-h6hf"
server_key: "6062fbe7574954bd91bc230d3e5a1b76"
server_key_iv: "0428602710f4f9998695e4a9a78328c5"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:21.568717 8557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:21.595391 8557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:21.600140 8557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:21.600479 8557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:21.601677 8557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:21.603312 8557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:21.603387 8557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:21.603615 8557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:21.603705 8557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:21.638796 8557 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:40833
I20260925 01:32:21.638834 8684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:40833 every 8 connection(s)
I20260925 01:32:21.640235 8557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:32:21.640681 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 8557
I20260925 01:32:21.640883 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:32:21.641580 28288 external_mini_cluster.cc:1428] Setting key 4a48d1cd7d637e97bb9609271470315c
Sep 25 01:32:21 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299941, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:21.662385 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:21.643379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:42281 (local address 127.27.160.62:40105)
0925 01:32:21.644197 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:21.644203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:21.645199 (+ 996us) server_negotiation.cc:407] Connection header received
0925 01:32:21.647236 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:21.647240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:21.647321 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:21.647423 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:21.648953 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.649785 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.651798 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.652222 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.652326 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:21.655990 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:21.655996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:21.655999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:21.656045 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:21.659021 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.659826 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.659832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.659835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.660019 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.660562 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.660566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.660568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.660879 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:21.661007 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:21.661879 (+ 872us) server_negotiation.cc:299] Negotiation successful
0925 01:32:21.662095 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":300,"threads_started":1}
I20260925 01:32:21.663471 8688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:21.644083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40105 (local address 127.27.160.3:42281)
0925 01:32:21.645002 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:21.645076 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:21.646963 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:21.647657 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:21.647666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:21.648158 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:21.648762 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:21.648780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.650003 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.650007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:21.651639 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:21.651658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.652445 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.652451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:21.652670 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:21.653491 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:21.653520 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:21.655671 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:21.659168 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:21.659176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:21.659191 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:21.659671 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:21.660156 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:21.660159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:21.660162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:21.660442 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:21.661020 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:21.661027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:21.661497 (+ 470us) client_negotiation.cc:769] Sending connection context
0925 01:32:21.661762 (+ 265us) client_negotiation.cc:240] Negotiation successful
0925 01:32:21.662173 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":367,"threads_started":1}
I20260925 01:32:21.665365 8685 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40105
I20260925 01:32:21.665731 8685 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:21.666664 8685 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
I20260925 01:32:21.668450 8195 ts_manager.cc:194] Registered new tserver with Master: 82741e5dfc904ba9b5799c074034b2f4 (127.27.160.3:40833)
I20260925 01:32:21.669685 8195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:42281
I20260925 01:32:21.671619 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:32:21.673514 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 8275
Sep 25 01:32:21 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:32:21.693274 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:21.674553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.27.160.1:35071)
0925 01:32:21.675319 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:21.675327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:21.675353 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:32:21.675487 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:21.675494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:21.675758 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:21.675907 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:21.677738 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.679157 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.681047 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.681769 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.681957 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:21.686564 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:21.686580 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:21.686598 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:21.686641 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:21.690095 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.691026 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.691034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.691036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.691190 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.691599 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.691604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.691606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.692070 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:21.692336 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:21.692652 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 01:32:21.692875 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":431,"threads_started":1}
I20260925 01:32:21.694685 8403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:21.781277 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 8275
I20260925 01:32:21.798058 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 8414
Sep 25 01:32:21 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:32:21.817698 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:21.799132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.27.160.2:41937)
0925 01:32:21.799685 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:21.799692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:21.799762 (+ 70us) server_negotiation.cc:407] Connection header received
0925 01:32:21.799840 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:21.799847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:21.800087 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:21.800232 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:21.801928 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.803223 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.804847 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.805587 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.805715 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:21.810635 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:21.810655 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:21.810674 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:21.810811 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:21.814462 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.815303 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.815311 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.815314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.815484 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.815875 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.815879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.815881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.816272 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:21.816557 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:21.817016 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 01:32:21.817391 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":300,"threads_started":1}
I20260925 01:32:21.819180 8546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:21.902462 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 8414
I20260925 01:32:21.923552 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 8557
Sep 25 01:32:21 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:32:21.941365 8694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:21.924671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.27.160.3:40833)
0925 01:32:21.925393 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:21.925401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:21.925434 (+ 33us) server_negotiation.cc:407] Connection header received
0925 01:32:21.925496 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:21.925502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:21.925739 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:21.925879 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:21.927313 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.928353 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.929792 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:21.930554 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:21.930702 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:21.935154 (+ 4452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:21.935170 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:21.935186 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:21.935226 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:21.938448 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.939188 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.939196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.939199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.939368 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:21.939745 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:21.939751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:21.939754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:21.940132 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:21.940422 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:21.940657 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 01:32:21.940859 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":323,"threads_started":1}
I20260925 01:32:21.942715 8680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:22.030390 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 8557
I20260925 01:32:22.048478 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 8163
I20260925 01:32:22.050086 8225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:22.158524 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 8163
I20260925 01:32:22.180984 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40105
--webserver_interface=127.27.160.62
--webserver_port=37627
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:40105
--ranger_config_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:36869/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.160.20:36869/kms with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:22.526944 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:22.527280 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:22.527446 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:22.539929 8697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:22.540095 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:22.540198 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:22.540292 8697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:22.540365 8697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:22.557933 8697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.160.20:36869/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:40105
--ranger_config_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=37627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:22.559484 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:22.560964 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:22.575299 8705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:22.575338 8703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:22.576148 8702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:22.579084 8697 server_base.cc:1034] running on GCE node
I20260925 01:32:22.580147 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:22.581980 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:22.583244 8697 hybrid_clock.cc:648] HybridClock initialized: now 1790299942583218 us; error 125 us; skew 500 ppm
I20260925 01:32:22.583654 8697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:22 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299942, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:22.587287 8697 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:32:22.589067 8697 webserver.cc:460] Webserver started at http://127.27.160.62:37627/ using document root <none> and password file <none>
I20260925 01:32:22.590157 8697 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:22.590309 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:32:22 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299942, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:22.661639 8697 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 01:32:22.665198 8713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:22.667049 8697 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:32:22.667280 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "607d2734698f47b196d4921673d80dc0"
format_stamp: "Formatted at 2026-09-25 01:32:18 on dist-test-slave-h6hf"
server_key: "5cbae6a78f0ea3c6d5c478972cd6a7ab"
server_key_iv: "f40b7796a3217ea591bb981edb08afe0"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:22.668335 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:22.696739 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:22.701254 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:22.701591 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:22.722259 8697 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:40105
I20260925 01:32:22.722266 8764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:40105 every 8 connection(s)
I20260925 01:32:22.723866 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:32:22.727679 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 8697
I20260925 01:32:22.728531 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:36869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:35071
--local_ip_for_outbound_sockets=127.27.160.1
--tserver_master_addrs=127.27.160.62:40105
--webserver_port=44271
--webserver_interface=127.27.160.1
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.160.20:36869/kms with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:32:22.732573 8765 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:32:22.736809 8765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Bootstrap starting.
I20260925 01:32:22.759204 8765 log.cc:824] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:22.770524 8765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:32:22.771212 8765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Bootstrap complete.
I20260925 01:32:22.782591 8765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } }
I20260925 01:32:22.783674 8765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607d2734698f47b196d4921673d80dc0, State: Initialized, Role: FOLLOWER
I20260925 01:32:22.784576 8765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } }
I20260925 01:32:22.784888 8765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:32:22.785048 8765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:32:22.785336 8765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:32:22.790215 8765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } }
I20260925 01:32:22.790876 8765 leader_election.cc:302] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 607d2734698f47b196d4921673d80dc0; no voters:
I20260925 01:32:22.791675 8765 leader_election.cc:288] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:32:22.791874 8769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:32:22.793087 8769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [term 2 LEADER]: Becoming Leader. State: Replica: 607d2734698f47b196d4921673d80dc0, State: Running, Role: LEADER
I20260925 01:32:22.793910 8769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } }
I20260925 01:32:22.794430 8765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:32:22.797139 8770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "607d2734698f47b196d4921673d80dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } } }
I20260925 01:32:22.797730 8770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:22.802163 8771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 607d2734698f47b196d4921673d80dc0. Latest consensus state: current_term: 2 leader_uuid: "607d2734698f47b196d4921673d80dc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607d2734698f47b196d4921673d80dc0" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40105 } } }
I20260925 01:32:22.803532 8771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:22.805681 8775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:32:22.811949 8775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:32:22.813552 8775 catalog_manager.cc:1259] Loaded cluster ID: 8b78e2ce5dbf43d2914912f7297a494e
I20260925 01:32:22.813728 8775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:32:22.818553 8775 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:32:22.820380 8775 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Loaded TSK: 0
I20260925 01:32:22.822075 8775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:32:22.827066 8765 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 01:32:23.270979 8767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:23.271513 8767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:23.271708 8767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:23.283601 8767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:23.283869 8767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:23.284111 8767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:32:23.300418 8767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.160.20:36869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:35071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=44271
--tserver_master_addrs=127.27.160.62:40105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:23.302517 8767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:23.304332 8767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:23.351346 8803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:23.357998 8806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:23.358918 8767 server_base.cc:1034] running on GCE node
W20260925 01:32:23.361491 8802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:23.362659 8767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:23.373464 8767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:23.391022 8767 hybrid_clock.cc:648] HybridClock initialized: now 1790299943390889 us; error 159 us; skew 500 ppm
I20260925 01:32:23.391870 8767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:23 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:23.397152 8767 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:32:23.405426 8767 webserver.cc:460] Webserver started at http://127.27.160.1:44271/ using document root <none> and password file <none>
I20260925 01:32:23.406230 8767 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:23.406418 8767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:32:23 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:23.437368 8767 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.006s sys 0.005s
I20260925 01:32:23.441687 8813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:23.443877 8767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:32:23.444094 8767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "321fe3e175934380a3126cb1018c4aa7"
format_stamp: "Formatted at 2026-09-25 01:32:19 on dist-test-slave-h6hf"
server_key: "0a4793cf5f52559e2e76c736f5f7c0e2"
server_key_iv: "d479dcb70ff4b3cef149dc866ef85473"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:23.445770 8767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:23.468441 8767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:23.474663 8767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:23.475095 8767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:23.476552 8767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:23.478567 8767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:23.478724 8767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 01:32:23.478878 8767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:23.478992 8767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:23.613652 8767 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:35071
I20260925 01:32:23.613756 8925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:35071 every 8 connection(s)
I20260925 01:32:23.615900 8767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:32:23.620771 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 8767
I20260925 01:32:23.621668 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:36869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:41937
--local_ip_for_outbound_sockets=127.27.160.2
--tserver_master_addrs=127.27.160.62:40105
--webserver_port=35763
--webserver_interface=127.27.160.2
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.160.20:36869/kms with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:32:23 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:23.667042 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:23.619752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40105 (local address 127.27.160.1:38013)
0925 01:32:23.633190 (+ 13438us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:23.633265 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:23.636901 (+ 3636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:23.639701 (+ 2800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:23.639717 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:23.640425 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:23.641318 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:23.641350 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:23.644163 (+ 2813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:23.644171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:23.646606 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:23.646632 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:23.650583 (+ 3951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:23.650592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:23.651035 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:23.652011 (+ 976us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:23.652057 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:23.655399 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:23.661351 (+ 5952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:23.661365 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:23.661391 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:23.662114 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:23.662756 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:23.662761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:23.662766 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:23.663058 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:23.664170 (+ 1112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:23.664179 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:23.664635 (+ 456us) client_negotiation.cc:769] Sending connection context
0925 01:32:23.664950 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 01:32:23.665600 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12934,"thread_start_us":13586,"threads_started":1}
I20260925 01:32:23.669548 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:23.620054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:38013 (local address 127.27.160.62:40105)
0925 01:32:23.638154 (+ 18100us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:23.638171 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:23.638218 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:32:23.638613 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:23.638629 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:23.639039 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:23.639509 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:23.641616 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:23.643909 (+ 2293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:23.649396 (+ 5487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:23.650226 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:23.650541 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:23.655896 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:23.655915 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:23.655945 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:23.656018 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:23.660981 (+ 4963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:23.662340 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:23.662353 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:23.662363 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:23.662600 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:23.663195 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:23.663201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:23.663204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:23.663888 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:23.664092 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:23.667819 (+ 3727us) server_negotiation.cc:299] Negotiation successful
0925 01:32:23.668307 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17811,"thread_start_us":17502,"threads_started":1}
I20260925 01:32:23.673435 8926 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40105
I20260925 01:32:23.673837 8926 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:23.674855 8926 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
W20260925 01:32:23.678733 8926 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:40105 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:32:23.679006 8926 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:23.679344 8926 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
I20260925 01:32:23.681010 8926 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:23.681373 8926 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
W20260925 01:32:23.682723 8926 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:32:24.082854 8930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:24.083392 8930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:24.083567 8930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:24.095515 8930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:24.095669 8930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:24.095862 8930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:32:24.112182 8930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.160.20:36869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:41937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=35763
--tserver_master_addrs=127.27.160.62:40105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:24.114207 8930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:24.116070 8930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:24.133828 8943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:24.134735 8944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:24.134797 8946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:24.135226 8930 server_base.cc:1034] running on GCE node
I20260925 01:32:24.136067 8930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:24.137439 8930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:24.138702 8930 hybrid_clock.cc:648] HybridClock initialized: now 1790299944138630 us; error 57 us; skew 500 ppm
I20260925 01:32:24.139123 8930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:24 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:24.143471 8930 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:32:24.145452 8930 webserver.cc:460] Webserver started at http://127.27.160.2:35763/ using document root <none> and password file <none>
I20260925 01:32:24.146467 8930 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:24.146674 8930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:32:24 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:24.172739 8930 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 01:32:24.176844 8954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:24.178833 8930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260925 01:32:24.179061 8930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "05bc40d2999e493c8938f2ba8d5423e6"
format_stamp: "Formatted at 2026-09-25 01:32:20 on dist-test-slave-h6hf"
server_key: "e620852dd519efc8a6abc40e042093d3"
server_key_iv: "364d7b046f375b80fe7319a6d18e36a8"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:24.180606 8930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:24.230154 8930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:24.234740 8930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:24.235046 8930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:24.236191 8930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:24.237756 8930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:24.237881 8930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:24.238019 8930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:24.238106 8930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:24.336591 8930 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:41937
I20260925 01:32:24.336687 9070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:41937 every 8 connection(s)
I20260925 01:32:24.338809 8930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:32:24.347904 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 8930
I20260925 01:32:24.348831 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.160.20:36869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:40833
--local_ip_for_outbound_sockets=127.27.160.3
--tserver_master_addrs=127.27.160.62:40105
--webserver_port=37181
--webserver_interface=127.27.160.3
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.160.20:36869/kms with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:32:24 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:24.388620 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:24.343384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:44423 (local address 127.27.160.62:40105)
0925 01:32:24.366133 (+ 22749us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:24.366140 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:24.366165 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:32:24.366237 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:24.366242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:24.366350 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:24.366503 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:24.368923 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:24.370265 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:24.373171 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:24.373938 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:24.374065 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:24.379418 (+ 5353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:24.379427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:24.379432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:24.379482 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:24.383636 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:24.385085 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:24.385092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:24.385096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:24.385314 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:24.386010 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:24.386016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:24.386019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:24.386466 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:24.386613 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:24.387877 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0925 01:32:24.388162 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22593,"thread_start_us":22418,"threads_started":1}
I20260925 01:32:24.391801 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:24.342686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40105 (local address 127.27.160.2:44423)
0925 01:32:24.362007 (+ 19321us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:24.362089 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:24.364890 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:24.366933 (+ 2043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:24.366948 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:24.367707 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:24.368644 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:24.368686 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:24.370513 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:24.370520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:24.372928 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:24.372951 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:24.374511 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:24.374519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:24.374880 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:24.375822 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:24.375857 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:24.379045 (+ 3188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:24.384022 (+ 4977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:24.384036 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:24.384060 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:24.384891 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:24.385489 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:24.385494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:24.385498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:24.385848 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:24.386759 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:24.386776 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:24.387350 (+ 574us) client_negotiation.cc:769] Sending connection context
0925 01:32:24.389325 (+ 1975us) client_negotiation.cc:240] Negotiation successful
0925 01:32:24.390069 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18888,"thread_start_us":18607,"threads_started":1}
I20260925 01:32:24.394342 9071 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40105
I20260925 01:32:24.394733 9071 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:24.395803 9071 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
I20260925 01:32:24.398502 8730 ts_manager.cc:194] Registered new tserver with Master: 05bc40d2999e493c8938f2ba8d5423e6 (127.27.160.2:41937)
I20260925 01:32:24.401079 8730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:44423
Sep 25 01:32:24 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:24.686806 8926 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:24.687296 8926 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
I20260925 01:32:24.688805 8730 ts_manager.cc:194] Registered new tserver with Master: 321fe3e175934380a3126cb1018c4aa7 (127.27.160.1:35071)
I20260925 01:32:24.690011 8730 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:38013
Sep 25 01:32:24 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
W20260925 01:32:24.860263 9075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:24.860800 9075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:24.860991 9075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:24.874881 9075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:24.875114 9075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:24.875355 9075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:32:24.892465 9075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.160.20:36869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:40833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=37181
--tserver_master_addrs=127.27.160.62:40105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:24.894675 9075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:24.896644 9075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:24.925455 9081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:24.926628 9082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:24.933853 9075 server_base.cc:1034] running on GCE node
W20260925 01:32:24.934345 9085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:24.935252 9075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:24.936717 9075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:24.938054 9075 hybrid_clock.cc:648] HybridClock initialized: now 1790299944937971 us; error 78 us; skew 500 ppm
I20260925 01:32:24.938489 9075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:24 dist-test-slave-h6hf krb5kdc[7719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:24.943277 9075 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:32:24.945642 9075 webserver.cc:460] Webserver started at http://127.27.160.3:37181/ using document root <none> and password file <none>
I20260925 01:32:24.946734 9075 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:24.946947 9075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:32:24 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:24.993039 9075 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.006s sys 0.005s
I20260925 01:32:25.019893 9092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:25.022531 9075 fs_manager.cc:725] Time spent opening block manager: real 0.027s user 0.018s sys 0.007s
I20260925 01:32:25.022888 9075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "82741e5dfc904ba9b5799c074034b2f4"
format_stamp: "Formatted at 2026-09-25 01:32:21 on dist-test-slave-h6hf"
server_key: "6062fbe7574954bd91bc230d3e5a1b76"
server_key_iv: "0428602710f4f9998695e4a9a78328c5"
server_key_version: "kuduclusterkey@0"
I20260925 01:32:25.024541 9075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:25.127202 9075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:25.137060 9075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:25.138301 9075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:25.147823 9075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:25.156140 9075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:25.156375 9075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 01:32:25.156548 9075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:25.156656 9075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:32:25 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for HTTP/127.27.160.20@KRBTEST.COM
I20260925 01:32:25.407328 9071 heartbeater.cc:503] Master 127.27.160.62:40105 was elected leader, sending a full tablet report...
I20260925 01:32:25.526458 9075 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:40833
I20260925 01:32:25.526507 9205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:40833 every 8 connection(s)
I20260925 01:32:25.528580 9075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:32:25.535296 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 9075
I20260925 01:32:25.623939 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.544241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.27.160.62:40105)
0925 01:32:25.599318 (+ 55077us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:25.599324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:25.599344 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:32:25.599399 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:25.599405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:25.599473 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:25.599576 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:25.602097 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.603259 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.605536 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.606228 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.606357 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.612454 (+ 6097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:25.612462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:25.612467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:25.612514 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:25.619641 (+ 7127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:25.620733 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:25.620740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:25.620744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:25.620912 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:25.621420 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:25.621426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:25.621430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:25.621774 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:25.622070 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:25.623324 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0925 01:32:25.623564 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54909,"thread_start_us":52917,"threads_started":1}
Sep 25 01:32:25 dist-test-slave-h6hf krb5kdc[7719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:25.635926 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.533577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:58751 (local address 127.27.160.62:40105)
0925 01:32:25.592964 (+ 59387us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:25.592972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:25.598301 (+ 5329us) server_negotiation.cc:407] Connection header received
0925 01:32:25.598414 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:25.598420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:25.598544 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:25.598653 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:25.617286 (+ 18633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.618561 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.622078 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.622817 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.622934 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.627730 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:25.627737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:25.627742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:25.627784 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:25.632047 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:25.633239 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:25.633244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:25.633247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:25.633392 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:25.633963 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:25.633967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:25.633969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:25.634245 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:25.634366 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:25.635366 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0925 01:32:25.635560 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59231,"thread_start_us":71743,"threads_started":1}
I20260925 01:32:25.637531 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.547350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40105 (local address 127.27.160.3:58751)
0925 01:32:25.593753 (+ 46403us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:25.593834 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:25.596624 (+ 2790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:25.607975 (+ 11351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:25.607991 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:25.608746 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:25.615989 (+ 7243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.616027 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.618899 (+ 2872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.618907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.621606 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.621635 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.623380 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.623388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.623724 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.624589 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:25.624628 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:25.627454 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:25.632279 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:25.632290 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:25.632311 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:25.633014 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:25.633534 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:25.633539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:25.633543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:25.633836 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:25.634408 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:25.634417 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:25.634914 (+ 497us) client_negotiation.cc:769] Sending connection context
0925 01:32:25.635272 (+ 358us) client_negotiation.cc:240] Negotiation successful
0925 01:32:25.635846 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45977,"thread_start_us":49759,"threads_started":1}
I20260925 01:32:25.639555 9206 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40105
I20260925 01:32:25.639922 9206 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:25.640811 9206 heartbeater.cc:511] Master 127.27.160.62:40105 requested a full tablet report, sending...
I20260925 01:32:25.642437 8729 ts_manager.cc:194] Registered new tserver with Master: 82741e5dfc904ba9b5799c074034b2f4 (127.27.160.3:40833)
I20260925 01:32:25.643474 8729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:58751
I20260925 01:32:25.654174 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:32:25.674729 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.658319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.27.160.62:40105)
0925 01:32:25.658566 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:25.658575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:25.659046 (+ 471us) server_negotiation.cc:407] Connection header received
0925 01:32:25.659210 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:25.659216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:25.659339 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:25.659459 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:25.661349 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.662466 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.663938 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.664628 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.664774 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.667701 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:25.667708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:25.667713 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:25.667763 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:25.671536 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:25.672291 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:25.672297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:25.672301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:25.672464 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:25.672893 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:25.672899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:25.672902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:25.673349 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:25.673614 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:25.674114 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 01:32:25.674386 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:32:25.680440 8729 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:32:25.690573 8729 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:32:25.693567 8926 heartbeater.cc:503] Master 127.27.160.62:40105 was elected leader, sending a full tablet report...
I20260925 01:32:25.732542 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.709328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41937 (local address 127.0.0.1:38358)
0925 01:32:25.722688 (+ 13360us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:25.722743 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:25.722931 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:25.724343 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:25.724351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:25.724382 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:25.724825 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.724836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.727043 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.727048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.729449 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.729466 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.731593 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.731600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.731802 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.731858 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 01:32:25.731975 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 01:32:25.732169 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13051,"threads_started":1}
I20260925 01:32:25.733639 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.709501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.27.160.2:41937)
0925 01:32:25.723676 (+ 14175us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:25.723685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:25.723715 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:32:25.723816 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:25.723824 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:25.724094 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:25.725054 (+ 960us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:25.725324 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.726838 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.729671 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.731361 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.732777 (+ 1416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.733035 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:25.733109 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 01:32:25.733311 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13989,"thread_start_us":13809,"threads_started":1}
I20260925 01:32:25.738871 9006 tablet_service.cc:1467] Processing CreateTablet for tablet a38c608fc4e44c33a90c458caa66012a (DEFAULT_TABLE table=test-table [id=27eec11d6df44ecf982cf07a2d62501c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:25.740689 9006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38c608fc4e44c33a90c458caa66012a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:25.747340 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.709981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:40833 (local address 127.0.0.1:44770)
0925 01:32:25.731055 (+ 21074us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:25.731075 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:25.731211 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:25.732306 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:25.732311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:25.732341 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:25.732749 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.732760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.737837 (+ 5077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.737848 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.741004 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.741035 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.746377 (+ 5342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.746384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.746579 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.746618 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:32:25.746729 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:32:25.746927 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20916,"thread_start_us":12245,"threads_started":1}
I20260925 01:32:25.748353 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.711004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.27.160.3:40833)
0925 01:32:25.730555 (+ 19551us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:25.730565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:25.731181 (+ 616us) server_negotiation.cc:407] Connection header received
0925 01:32:25.731369 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:25.731393 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:25.731688 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:25.732003 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:25.733062 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.737571 (+ 4509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.741441 (+ 3870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.745735 (+ 4294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.745931 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.746189 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:25.747796 (+ 1607us) server_negotiation.cc:299] Negotiation successful
0925 01:32:25.748006 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19295,"thread_start_us":19075,"threads_started":1}
I20260925 01:32:25.754379 9141 tablet_service.cc:1467] Processing CreateTablet for tablet a38c608fc4e44c33a90c458caa66012a (DEFAULT_TABLE table=test-table [id=27eec11d6df44ecf982cf07a2d62501c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:25.756029 9141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38c608fc4e44c33a90c458caa66012a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:25.765817 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.734108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:35071 (local address 127.0.0.1:45794)
0925 01:32:25.734248 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:25.734266 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:25.734359 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:25.746910 (+ 12551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:25.746915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:25.746949 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:25.747132 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.747143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.749393 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.749402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.750943 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.750957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.764887 (+ 13930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.764896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.765189 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.765249 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 01:32:25.765342 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 01:32:25.765514 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 01:32:25.767261 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.733825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.27.160.1:35071)
0925 01:32:25.746216 (+ 12391us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:25.746227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:25.746258 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:32:25.746397 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:25.746406 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:25.746689 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:25.746933 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:25.747418 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.749200 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.761071 (+ 11871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.764626 (+ 3555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.766345 (+ 1719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.766616 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:25.766701 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 01:32:25.766909 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12182,"thread_start_us":13307,"threads_started":1}
I20260925 01:32:25.771968 8861 tablet_service.cc:1467] Processing CreateTablet for tablet a38c608fc4e44c33a90c458caa66012a (DEFAULT_TABLE table=test-table [id=27eec11d6df44ecf982cf07a2d62501c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:25.774202 8861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38c608fc4e44c33a90c458caa66012a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:25.778324 9233 tablet_bootstrap.cc:492] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Bootstrap starting.
I20260925 01:32:25.783792 9234 tablet_bootstrap.cc:492] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Bootstrap starting.
I20260925 01:32:25.786144 9233 tablet_bootstrap.cc:654] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:25.787714 9234 tablet_bootstrap.cc:654] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:25.787906 9233 log.cc:824] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:25.788895 9234 log.cc:824] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:25.795336 9234 tablet_bootstrap.cc:492] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: No bootstrap required, opened a new log
I20260925 01:32:25.795629 9234 ts_tablet_manager.cc:1397] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260925 01:32:25.802268 9237 tablet_bootstrap.cc:492] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Bootstrap starting.
I20260925 01:32:25.804162 9233 tablet_bootstrap.cc:492] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: No bootstrap required, opened a new log
I20260925 01:32:25.804468 9233 ts_tablet_manager.cc:1397] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.008s
I20260925 01:32:25.806403 9237 tablet_bootstrap.cc:654] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:25.808454 9237 log.cc:824] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:25.812508 9234 raft_consensus.cc:348] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.813331 9234 raft_consensus.cc:374] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:25.813567 9234 raft_consensus.cc:729] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82741e5dfc904ba9b5799c074034b2f4, State: Initialized, Role: FOLLOWER
I20260925 01:32:25.814628 9234 consensus_queue.cc:260] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.815127 9233 raft_consensus.cc:348] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.815606 9233 raft_consensus.cc:374] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:25.815704 9233 raft_consensus.cc:729] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bc40d2999e493c8938f2ba8d5423e6, State: Initialized, Role: FOLLOWER
I20260925 01:32:25.816375 9233 consensus_queue.cc:260] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.822412 9206 heartbeater.cc:503] Master 127.27.160.62:40105 was elected leader, sending a full tablet report...
I20260925 01:32:25.823290 9234 ts_tablet_manager.cc:1428] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Time spent starting tablet: real 0.028s user 0.010s sys 0.011s
I20260925 01:32:25.824867 9237 tablet_bootstrap.cc:492] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: No bootstrap required, opened a new log
I20260925 01:32:25.825049 9233 ts_tablet_manager.cc:1428] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260925 01:32:25.825206 9237 ts_tablet_manager.cc:1397] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.009s
I20260925 01:32:25.836383 9237 raft_consensus.cc:348] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.836930 9237 raft_consensus.cc:374] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:25.837044 9237 raft_consensus.cc:729] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321fe3e175934380a3126cb1018c4aa7, State: Initialized, Role: FOLLOWER
I20260925 01:32:25.837841 9237 consensus_queue.cc:260] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.839819 9237 ts_tablet_manager.cc:1428] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260925 01:32:25.847817 9072 tablet.cc:2369] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:32:25.864506 9207 tablet.cc:2369] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:32:25.882054 8927 tablet.cc:2369] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:32:25.941368 9239 raft_consensus.cc:482] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:32:25.941789 9239 raft_consensus.cc:504] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.943801 9239 leader_election.cc:288] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05bc40d2999e493c8938f2ba8d5423e6 (127.27.160.2:41937), 321fe3e175934380a3126cb1018c4aa7 (127.27.160.1:35071)
I20260925 01:32:25.950773 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.944168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:55583 (local address 127.27.160.2:41937)
0925 01:32:25.944356 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:25.944362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:25.944398 (+ 36us) server_negotiation.cc:407] Connection header received
0925 01:32:25.944651 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:25.944660 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:25.944769 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:25.944879 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:25.945435 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.946438 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.948633 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.949838 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.949997 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.950069 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:25.950393 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 01:32:25.950525 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:32:25.950812 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.944043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41937 (local address 127.27.160.3:55583)
0925 01:32:25.944271 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:25.944295 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:25.944437 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:25.944939 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:25.944944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:25.944977 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:25.945226 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.945238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.946597 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.946606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.948446 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.948461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.949975 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.949983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.950190 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.950232 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 01:32:25.950348 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 01:32:25.950502 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 01:32:25.952312 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.944486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:35071 (local address 127.27.160.3:55537)
0925 01:32:25.945533 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:25.945549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:25.945699 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:25.946098 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:25.946103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:25.946131 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:25.946475 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.946484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.947792 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.947800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.949725 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:25.949738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.951548 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.951554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:25.951714 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.951746 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:32:25.951874 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 01:32:25.951999 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":720,"threads_started":1}
I20260925 01:32:25.952428 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:25.944851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:55537 (local address 127.27.160.1:35071)
0925 01:32:25.945103 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:25.945111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:25.945646 (+ 535us) server_negotiation.cc:407] Connection header received
0925 01:32:25.945865 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:25.945870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:25.945932 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:25.946062 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:25.946681 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.947628 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.949888 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:25.951319 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:25.951501 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:25.951580 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:25.951946 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 01:32:25.952094 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:32:25.952394 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38c608fc4e44c33a90c458caa66012a" candidate_uuid: "82741e5dfc904ba9b5799c074034b2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bc40d2999e493c8938f2ba8d5423e6" is_pre_election: true
I20260925 01:32:25.953094 9026 raft_consensus.cc:2454] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82741e5dfc904ba9b5799c074034b2f4 in term 0.
I20260925 01:32:25.953714 8873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38c608fc4e44c33a90c458caa66012a" candidate_uuid: "82741e5dfc904ba9b5799c074034b2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321fe3e175934380a3126cb1018c4aa7" is_pre_election: true
I20260925 01:32:25.954042 9094 leader_election.cc:302] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bc40d2999e493c8938f2ba8d5423e6, 82741e5dfc904ba9b5799c074034b2f4; no voters:
I20260925 01:32:25.954239 8873 raft_consensus.cc:2454] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82741e5dfc904ba9b5799c074034b2f4 in term 0.
I20260925 01:32:25.954536 9239 raft_consensus.cc:2781] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:32:25.954697 9239 raft_consensus.cc:482] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:32:25.954846 9239 raft_consensus.cc:3037] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:25.957487 9239 raft_consensus.cc:504] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.958105 9239 leader_election.cc:288] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [CANDIDATE]: Term 1 election: Requested vote from peers 05bc40d2999e493c8938f2ba8d5423e6 (127.27.160.2:41937), 321fe3e175934380a3126cb1018c4aa7 (127.27.160.1:35071)
I20260925 01:32:25.958731 9026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38c608fc4e44c33a90c458caa66012a" candidate_uuid: "82741e5dfc904ba9b5799c074034b2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05bc40d2999e493c8938f2ba8d5423e6"
I20260925 01:32:25.958961 9026 raft_consensus.cc:3037] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:25.958940 8873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38c608fc4e44c33a90c458caa66012a" candidate_uuid: "82741e5dfc904ba9b5799c074034b2f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321fe3e175934380a3126cb1018c4aa7"
I20260925 01:32:25.959172 8873 raft_consensus.cc:3037] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:25.961979 8873 raft_consensus.cc:2454] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82741e5dfc904ba9b5799c074034b2f4 in term 1.
I20260925 01:32:25.962123 9026 raft_consensus.cc:2454] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82741e5dfc904ba9b5799c074034b2f4 in term 1.
I20260925 01:32:25.962669 9094 leader_election.cc:302] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bc40d2999e493c8938f2ba8d5423e6, 82741e5dfc904ba9b5799c074034b2f4; no voters:
I20260925 01:32:25.963042 9239 raft_consensus.cc:2781] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:32:25.963658 9239 raft_consensus.cc:686] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 1 LEADER]: Becoming Leader. State: Replica: 82741e5dfc904ba9b5799c074034b2f4, State: Running, Role: LEADER
I20260925 01:32:25.964265 9239 consensus_queue.cc:237] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } }
I20260925 01:32:25.971526 8729 catalog_manager.cc:5495] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82741e5dfc904ba9b5799c074034b2f4 (127.27.160.3). New cstate: current_term: 1 leader_uuid: "82741e5dfc904ba9b5799c074034b2f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82741e5dfc904ba9b5799c074034b2f4" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40833 } health_report { overall_health: HEALTHY } } }
I20260925 01:32:26.026789 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:26.020951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.27.160.3:40833)
0925 01:32:26.021163 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:26.021169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:26.021275 (+ 106us) server_negotiation.cc:407] Connection header received
0925 01:32:26.021556 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:26.021560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:26.021653 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:26.021828 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:32:26.022255 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:26.022987 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:26.024429 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:26.024880 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:26.025026 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:26.025621 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:32:26.026168 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:32:26.026372 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:26.026424 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 01:32:26.026510 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:32:26.035800 9026 raft_consensus.cc:1261] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 1 FOLLOWER]: Refusing update from remote peer 82741e5dfc904ba9b5799c074034b2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:32:26.035787 8873 raft_consensus.cc:1261] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 1 FOLLOWER]: Refusing update from remote peer 82741e5dfc904ba9b5799c074034b2f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:32:26.036870 9243 consensus_queue.cc:1035] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "321fe3e175934380a3126cb1018c4aa7" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 35071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:32:26.037339 9239 consensus_queue.cc:1035] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05bc40d2999e493c8938f2ba8d5423e6" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:32:26.065659 8730 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:32:26.065969 8730 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:32:26.070274 8730 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 607d2734698f47b196d4921673d80dc0: Sending DeleteTablet for 3 replicas of tablet a38c608fc4e44c33a90c458caa66012a
I20260925 01:32:26.071702 8861 tablet_service.cc:1514] Processing DeleteTablet for tablet a38c608fc4e44c33a90c458caa66012a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:26 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:45794
I20260925 01:32:26.071724 9141 tablet_service.cc:1514] Processing DeleteTablet for tablet a38c608fc4e44c33a90c458caa66012a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:26 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:44770
I20260925 01:32:26.072120 9006 tablet_service.cc:1514] Processing DeleteTablet for tablet a38c608fc4e44c33a90c458caa66012a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:26 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:38358
I20260925 01:32:26.081833 9258 tablet_replica.cc:330] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: stopping tablet replica
I20260925 01:32:26.083245 9258 raft_consensus.cc:2229] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:32:26.084261 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 8767
I20260925 01:32:26.084895 9258 raft_consensus.cc:2258] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:26.086364 9257 tablet_replica.cc:330] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: stopping tablet replica
I20260925 01:32:26.087445 9256 tablet_replica.cc:330] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: stopping tablet replica
I20260925 01:32:26.087904 9256 raft_consensus.cc:2229] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:32:26.088754 9256 raft_consensus.cc:2258] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:26.086860 9257 raft_consensus.cc:2229] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:32:26.089239 9258 ts_tablet_manager.cc:1905] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:26.089610 9257 raft_consensus.cc:2258] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:26.092110 9257 ts_tablet_manager.cc:1905] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:26.093660 9256 ts_tablet_manager.cc:1905] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:26.100072 9257 ts_tablet_manager.cc:1918] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:26.100287 9257 log.cc:1196] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/a38c608fc4e44c33a90c458caa66012a
I20260925 01:32:26.101080 9257 ts_tablet_manager.cc:1939] T a38c608fc4e44c33a90c458caa66012a P 321fe3e175934380a3126cb1018c4aa7: Deleting consensus metadata
I20260925 01:32:26.103289 8715 catalog_manager.cc:4841] TS 321fe3e175934380a3126cb1018c4aa7 (127.27.160.1:35071): tablet a38c608fc4e44c33a90c458caa66012a (table test-table [id=27eec11d6df44ecf982cf07a2d62501c]) successfully deleted
I20260925 01:32:26.104696 9258 ts_tablet_manager.cc:1918] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:26.104902 9258 log.cc:1196] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/a38c608fc4e44c33a90c458caa66012a
I20260925 01:32:26.105489 9258 ts_tablet_manager.cc:1939] T a38c608fc4e44c33a90c458caa66012a P 05bc40d2999e493c8938f2ba8d5423e6: Deleting consensus metadata
I20260925 01:32:26.106084 9256 ts_tablet_manager.cc:1918] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:26.106336 9256 log.cc:1196] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/a38c608fc4e44c33a90c458caa66012a
I20260925 01:32:26.106986 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:26.086165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.27.160.1:35071)
0925 01:32:26.086453 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:26.086459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:26.086478 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:32:26.086544 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:26.086550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:26.086627 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:26.086732 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:26.088666 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:26.089910 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:26.091898 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:26.092556 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:26.092691 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:26.096704 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:26.096720 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:26.096742 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:26.096800 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:26.101515 (+ 4715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:26.102646 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:26.102657 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:26.102661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:26.102838 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:26.104059 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:26.104065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:26.104069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:26.104586 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:26.104899 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:26.106262 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0925 01:32:26.106572 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260925 01:32:26.107625 9256 ts_tablet_manager.cc:1939] T a38c608fc4e44c33a90c458caa66012a P 82741e5dfc904ba9b5799c074034b2f4: Deleting consensus metadata
I20260925 01:32:26.107752 8921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:26.107872 8715 catalog_manager.cc:4841] TS 05bc40d2999e493c8938f2ba8d5423e6 (127.27.160.2:41937): tablet a38c608fc4e44c33a90c458caa66012a (table test-table [id=27eec11d6df44ecf982cf07a2d62501c]) successfully deleted
I20260925 01:32:26.110399 8717 catalog_manager.cc:4841] TS 82741e5dfc904ba9b5799c074034b2f4 (127.27.160.3:40833): tablet a38c608fc4e44c33a90c458caa66012a (table test-table [id=27eec11d6df44ecf982cf07a2d62501c]) successfully deleted
I20260925 01:32:26.201609 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 8767
I20260925 01:32:26.220315 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 8930
I20260925 01:32:26.237214 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:26.221290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.27.160.2:41937)
0925 01:32:26.221503 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:26.221509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:26.221533 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:32:26.221602 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:26.221606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:26.221689 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:26.221816 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:26.223210 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:26.224291 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:26.225768 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:26.226414 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:26.226546 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:26.229600 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:26.229621 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:26.229643 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:26.229734 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:26.233893 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:26.234728 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:26.234739 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:26.234743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:26.234908 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:26.235386 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:26.235391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:26.235395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:26.235889 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:26.236200 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:26.236522 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 01:32:26.236781 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:32:26.237965 9066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:26.331951 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 8930
I20260925 01:32:26.350543 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 9075
I20260925 01:32:26.366199 9231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:26.351479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.27.160.3:40833)
0925 01:32:26.351655 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:26.351663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:26.351690 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:32:26.351970 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:26.351974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:26.352069 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:26.352192 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:26.353667 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:26.354453 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:26.355982 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:26.356468 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:26.356616 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:26.359475 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:26.359489 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:26.359505 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:26.359544 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:26.362753 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:26.363556 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:26.363564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:26.363567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:26.363737 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:26.364170 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:26.364175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:26.364177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:26.364560 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:26.364919 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:26.365533 (+ 614us) server_negotiation.cc:299] Negotiation successful
0925 01:32:26.365762 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:32:26.367111 9201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:26.490916 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 9075
I20260925 01:32:26.510751 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 8697
I20260925 01:32:26.512136 8760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:26.627207 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 8697
I20260925 01:32:26.650108 28288 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 01:32:26.988257 28288 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 01:32:26.988483 28288 mini_ranger.cc:66] Stopping Ranger...
I20260925 01:32:27.334877 28288 mini_ranger.cc:68] Stopped Ranger
2026-09-25 01:32:27.335 UTC [7749] LOG: received smart shutdown request
2026-09-25 01:32:27.338 UTC [7749] LOG: background worker "logical replication launcher" (PID 7758) exited with exit code 1
2026-09-25 01:32:27.339 UTC [7753] LOG: shutting down
2026-09-25 01:32:27.529 UTC [7749] LOG: database system is shut down
2026-09-25T01:32:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55334 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:32:27 dist-test-slave-h6hf krb5kdc[9271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:32:27 dist-test-slave-h6hf krb5kdc[9271](info): set up 2 sockets
Sep 25 01:32:27 dist-test-slave-h6hf krb5kdc[9271](info): commencing operation
krb5kdc: starting...
W20260925 01:32:28.782755 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:32:28 dist-test-slave-h6hf krb5kdc[9271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:32:29.041755 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37097
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:37097,127.27.160.61:34293,127.27.160.60:43403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:29.390162 9284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:29.390486 9284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:29.390653 9284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:29.398286 9284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:29.398420 9284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:29.398504 9284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:29.398590 9284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:29.398669 9284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:29.409559 9284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:37097,127.27.160.61:34293,127.27.160.60:43403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:29.411087 9284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:29.412528 9284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:29.426079 9289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:29.426033 9292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:29.426096 9290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:29.427502 9284 server_base.cc:1034] running on GCE node
I20260925 01:32:29.428529 9284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:29.430292 9284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:29.431523 9284 hybrid_clock.cc:648] HybridClock initialized: now 1790299949431466 us; error 49 us; skew 500 ppm
I20260925 01:32:29.431946 9284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:29 dist-test-slave-h6hf krb5kdc[9271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:29.440330 9284 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:32:29.442158 9284 webserver.cc:460] Webserver started at http://127.27.160.62:39245/ using document root <none> and password file <none>
I20260925 01:32:29.442907 9284 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:29.443022 9284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:29.443281 9284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:32:29.446565 9284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "74b980fa101a447197fb77f2951d552b"
format_stamp: "Formatted at 2026-09-25 01:32:29 on dist-test-slave-h6hf"
server_key: "ba1621ad627ae5b4eb99c790277da3dc"
server_key_iv: "8ff416b784c1aca499043051687d1043"
server_key_version: "encryptionkey@0"
I20260925 01:32:29.447425 9284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "74b980fa101a447197fb77f2951d552b"
format_stamp: "Formatted at 2026-09-25 01:32:29 on dist-test-slave-h6hf"
server_key: "ba1621ad627ae5b4eb99c790277da3dc"
server_key_iv: "8ff416b784c1aca499043051687d1043"
server_key_version: "encryptionkey@0"
I20260925 01:32:29.453308 9284 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 01:32:29.457321 9299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:29.458851 9284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:32:29.459100 9284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "74b980fa101a447197fb77f2951d552b"
format_stamp: "Formatted at 2026-09-25 01:32:29 on dist-test-slave-h6hf"
server_key: "ba1621ad627ae5b4eb99c790277da3dc"
server_key_iv: "8ff416b784c1aca499043051687d1043"
server_key_version: "encryptionkey@0"
I20260925 01:32:29.459329 9284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:29.474824 9284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:29.479557 9284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:29.479884 9284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:29.499364 9284 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:37097
I20260925 01:32:29.499400 9350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:37097 every 8 connection(s)
I20260925 01:32:29.500905 9284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:32:29.505443 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:29.506994 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 9284
I20260925 01:32:29.507249 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:32:29.508096 28288 external_mini_cluster.cc:1428] Setting key 903c0b874850cf9ec1b3edba0d5789f6
I20260925 01:32:29.517477 9351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.160.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.61@KRBTEST.COM" created.
Sep 25 01:32:29 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:29.548914 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.521405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37097 (local address 127.0.0.1:53504)
0925 01:32:29.522464 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.522516 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:29.524729 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:29.525709 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:29.525716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:29.526482 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:29.526862 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:29.526886 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:29.528649 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:29.528664 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:29.530520 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:29.530552 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:29.532290 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:29.532298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:29.532563 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:29.533496 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:29.533541 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:29.538150 (+ 4609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:29.543547 (+ 5397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:29.543564 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:29.543580 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:29.544105 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:29.544644 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:29.544651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:29.544654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:29.544862 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:29.545907 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:29.545918 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:29.546208 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 01:32:29.546507 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 01:32:29.547276 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":170,"thread_start_us":514,"threads_started":1}
I20260925 01:32:29.548934 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.521405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53504 (local address 127.27.160.62:37097)
0925 01:32:29.523388 (+ 1983us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:29.523412 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:29.523472 (+ 60us) server_negotiation.cc:407] Connection header received
0925 01:32:29.525196 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:29.525202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:29.525464 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:29.525694 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:29.527080 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:29.528481 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:29.530709 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:29.531896 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:29.532184 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:29.538496 (+ 6312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:29.538510 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:29.538535 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:29.538593 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:29.543309 (+ 4716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:29.544286 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:29.544304 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:29.544306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:29.544501 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:29.545004 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:29.545009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:29.545013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:29.545727 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:29.545985 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:29.546682 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 01:32:29.547086 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":1737,"spinlock_wait_cycles":45312,"thread_start_us":1501,"threads_started":1}
I20260925 01:32:29.552918 9351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:32:29.554380 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.553877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55180)
0925 01:32:29.554096 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.554216 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260925 01:32:29.554816 9303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
I20260925 01:32:29.556790 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.556346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55182)
0925 01:32:29.556528 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.556612 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260925 01:32:29.557219 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:34293: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab.
Entry for principal kudu/127.27.160.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab.
I20260925 01:32:29.569552 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.61:34293
--webserver_interface=127.27.160.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab
--principal=kudu/127.27.160.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:37097,127.27.160.61:34293,127.27.160.60:43403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:32:29.606643 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } attempt: 1
I20260925 01:32:29.608141 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.607643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55192)
0925 01:32:29.607878 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.608009 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 01:32:29.609310 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.608858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55194)
0925 01:32:29.609014 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.609099 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:32:29.609508 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:34293: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
I20260925 01:32:29.677788 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } attempt: 2
I20260925 01:32:29.679056 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.678550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55210)
0925 01:32:29.678818 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.678911 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 01:32:29.680266 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.679737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55222)
0925 01:32:29.679942 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.680033 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260925 01:32:29.680480 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:34293: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
I20260925 01:32:29.771782 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } attempt: 3
I20260925 01:32:29.773083 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.772579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55226)
0925 01:32:29.772817 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.772943 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 01:32:29.774320 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.773896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55242)
0925 01:32:29.774056 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.774144 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:32:29.774535 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:34293: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
W20260925 01:32:29.915074 9358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:29.915493 9358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:29.915671 9358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 01:32:29.917845 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } attempt: 4
I20260925 01:32:29.919334 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.918742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55250)
0925 01:32:29.919069 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.919186 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260925 01:32:29.920460 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:29.920039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55260)
0925 01:32:29.920206 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:29.920310 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:32:29.920668 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:34293: Network error: Client connection negotiation failed: client connection to 127.27.160.61:34293: connect: Connection refused (error 111)
W20260925 01:32:29.923653 9358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:29.923804 9358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:29.923907 9358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:29.924012 9358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:29.924089 9358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:29.934279 9358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:37097,127.27.160.61:34293,127.27.160.60:43403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab
--principal=kudu/127.27.160.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.61:34293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.160.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:29.935729 9358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:29.937181 9358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:29.949864 9363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:29.949645 9364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:29.949879 9366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:29.951020 9358 server_base.cc:1034] running on GCE node
I20260925 01:32:29.951830 9358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:29.953469 9358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:29.954705 9358 hybrid_clock.cc:648] HybridClock initialized: now 1790299949954653 us; error 48 us; skew 500 ppm
I20260925 01:32:29.955116 9358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:29 dist-test-slave-h6hf krb5kdc[9271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:29.958473 9358 init.cc:377] Logged in from keytab as kudu/127.27.160.61@KRBTEST.COM (short username kudu)
I20260925 01:32:29.960193 9358 webserver.cc:460] Webserver started at http://127.27.160.61:39751/ using document root <none> and password file <none>
I20260925 01:32:29.960958 9358 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:29.961108 9358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:29.961504 9358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:32:29.964733 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/data/instance:
uuid: "777ebf47a30044308023282911789c95"
format_stamp: "Formatted at 2026-09-25 01:32:29 on dist-test-slave-h6hf"
server_key: "2737670f18071c10487577de8caef9e7"
server_key_iv: "cf883ee0f327e4abd85a59249ccd5c95"
server_key_version: "encryptionkey@0"
I20260925 01:32:29.965629 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/wal/instance:
uuid: "777ebf47a30044308023282911789c95"
format_stamp: "Formatted at 2026-09-25 01:32:29 on dist-test-slave-h6hf"
server_key: "2737670f18071c10487577de8caef9e7"
server_key_iv: "cf883ee0f327e4abd85a59249ccd5c95"
server_key_version: "encryptionkey@0"
I20260925 01:32:29.971169 9358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 01:32:29.974968 9373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:29.976475 9358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:32:29.976673 9358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/wal
uuid: "777ebf47a30044308023282911789c95"
format_stamp: "Formatted at 2026-09-25 01:32:29 on dist-test-slave-h6hf"
server_key: "2737670f18071c10487577de8caef9e7"
server_key_iv: "cf883ee0f327e4abd85a59249ccd5c95"
server_key_version: "encryptionkey@0"
I20260925 01:32:29.976922 9358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:30.003466 9358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:30.008348 9358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:30.008683 9358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:30.028733 9358 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.61:34293
I20260925 01:32:30.028827 9424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.61:34293 every 8 connection(s)
I20260925 01:32:30.030309 9358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
I20260925 01:32:30.034404 9425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:30.034770 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 9358
I20260925 01:32:30.034977 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-1/wal/instance
I20260925 01:32:30.035403 28288 external_mini_cluster.cc:1428] Setting key 0d1d4d25322d363a625f5df4a684d3cd
I20260925 01:32:30.041543 9425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.160.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.60@KRBTEST.COM" created.
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:30.066210 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.044594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.27.160.62:37097)
0925 01:32:30.044798 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.044803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.046123 (+ 1320us) server_negotiation.cc:407] Connection header received
0925 01:32:30.048196 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.048201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.048298 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.048416 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.050134 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.050835 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.052447 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.052892 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.053031 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.059616 (+ 6585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.059622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.059625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.059665 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.062637 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.063619 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.063624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.063626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.063760 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.064283 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.064293 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.064295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.064596 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.064764 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.065620 (+ 856us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.065875 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:32:30.067252 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.045090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37097 (local address 127.0.0.1:53516)
0925 01:32:30.045946 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.045992 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:30.047889 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:30.048684 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:30.048692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:30.049313 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:30.049942 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.049962 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.051012 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.051017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.052291 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.052304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.053198 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.053206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.053462 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.054188 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:30.054212 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:30.059321 (+ 5109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:30.062851 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.062862 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.062882 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.063466 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.063881 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.063887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.063891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.064104 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.064746 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:30.064755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:30.065205 (+ 450us) client_negotiation.cc:769] Sending connection context
0925 01:32:30.065505 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 01:32:30.066081 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":353,"threads_started":1}
I20260925 01:32:30.069183 9425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:32:30.084939 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.069882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55274)
0925 01:32:30.070041 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.070065 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:30.070166 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:30.071303 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:30.071307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:30.071767 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:30.071897 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.071905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.073321 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.073329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.074176 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.074187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.075007 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.075011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.075283 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.075874 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:30.075891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:30.078187 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:30.082199 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.082204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.082209 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.082627 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.083091 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.083094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.083096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.083306 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.083965 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:30.083969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:30.084096 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 01:32:30.084228 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 01:32:30.084531 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 01:32:30.084945 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.070164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55274 (local address 127.27.160.61:34293)
0925 01:32:30.070868 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.070874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.070900 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:32:30.070945 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.070951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.071139 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.071310 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.072053 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.073105 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.074313 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.074865 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.074998 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.078420 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.078433 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.078448 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.078477 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.082027 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.082834 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.082841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.082843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.082976 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.083446 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.083449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.083451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.083792 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.084042 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.084373 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.084647 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":543,"thread_start_us":408,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab.
Entry for principal kudu/127.27.160.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab.
I20260925 01:32:30.087455 9425 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:32:30.088635 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.60:43403
--webserver_interface=127.27.160.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab
--principal=kudu/127.27.160.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:37097,127.27.160.61:34293,127.27.160.60:43403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:32:30.088801 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.088340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:60736)
0925 01:32:30.088529 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.088656 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 01:32:30.089236 9376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
I20260925 01:32:30.090791 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.090480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:60746)
0925 01:32:30.090613 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.090686 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 01:32:30.091096 9425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
I20260925 01:32:30.140425 9425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 1
I20260925 01:32:30.141942 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.141220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:60754)
0925 01:32:30.141508 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.141679 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 01:32:30.143216 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.142741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:60758)
0925 01:32:30.142943 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.143044 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 01:32:30.143515 9425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
I20260925 01:32:30.211779 9425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 2
I20260925 01:32:30.213244 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.212542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:60770)
0925 01:32:30.212843 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.212950 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 01:32:30.214443 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.214041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:60780)
0925 01:32:30.214221 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.214300 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 01:32:30.214677 9425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
I20260925 01:32:30.219985 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } attempt: 5
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:32:30.234380 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.221065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.27.160.61:34293)
0925 01:32:30.221300 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.221304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.221319 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:32:30.221370 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.221373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.221440 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.221550 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.222367 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.223128 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.224326 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.224794 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.224939 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.228907 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.228912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.228916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.228952 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.231609 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.232430 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.232435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.232438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.232587 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.233020 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.233023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.233025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.233372 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.233501 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.233988 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.234164 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 01:32:30.234732 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.220728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:55288)
0925 01:32:30.220933 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.220960 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:30.221069 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:30.221586 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:30.221591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:30.222002 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:30.222183 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.222192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.223270 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.223275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.224173 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.224185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.224959 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.224965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.225212 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.225910 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:30.225928 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:30.228651 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:30.231790 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.231795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.231799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.232293 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.232720 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.232726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.232732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.232909 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.233520 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:30.233527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:30.233728 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 01:32:30.233892 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 01:32:30.234233 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 01:32:30.236117 9351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:32:30.237479 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.236922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:60796)
0925 01:32:30.237178 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.237291 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 01:32:30.238723 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.238366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:60808)
0925 01:32:30.238516 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.238607 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 01:32:30.239053 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
I20260925 01:32:30.290283 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 1
I20260925 01:32:30.291597 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.291076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54728)
0925 01:32:30.291313 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.291433 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 01:32:30.292649 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.292282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54738)
0925 01:32:30.292443 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.292530 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:32:30.292868 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
I20260925 01:32:30.305922 9425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 3
I20260925 01:32:30.307176 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.306665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54754)
0925 01:32:30.306918 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.307016 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 01:32:30.308517 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.308001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54760)
0925 01:32:30.308303 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.308389 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260925 01:32:30.308765 9425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
I20260925 01:32:30.361193 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 2
I20260925 01:32:30.362443 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.361906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54764)
0925 01:32:30.362165 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.362255 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:32:30.363590 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.363193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54778)
0925 01:32:30.363386 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.363461 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:32:30.363826 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
W20260925 01:32:30.443987 9432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:30.444360 9432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:30.444526 9432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:30.451897 9432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:30.452067 9432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260925 01:32:30.451996 9425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 4
W20260925 01:32:30.452181 9432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:30.452293 9432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:30.452382 9432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:30.453300 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.452736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54794)
0925 01:32:30.452971 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.453068 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 01:32:30.454470 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.454081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54802)
0925 01:32:30.454270 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.454347 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 01:32:30.454704 9425 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
I20260925 01:32:30.463805 9432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:37097,127.27.160.61:34293,127.27.160.60:43403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab
--principal=kudu/127.27.160.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.60:43403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.160.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:30.465320 9432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:30.466688 9432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:32:30.470058 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 3
I20260925 01:32:30.471181 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.470721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54812)
0925 01:32:30.470938 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.471027 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 01:32:30.472363 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.471902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54826)
0925 01:32:30.472145 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.472235 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260925 01:32:30.472575 9351 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:43403: Network error: Client connection negotiation failed: client connection to 127.27.160.60:43403: connect: Connection refused (error 111)
W20260925 01:32:30.479779 9440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:30.479894 9437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:30.479813 9438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:30.480733 9432 server_base.cc:1034] running on GCE node
I20260925 01:32:30.481846 9432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:30.483376 9432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:30.484583 9432 hybrid_clock.cc:648] HybridClock initialized: now 1790299950484524 us; error 60 us; skew 500 ppm
I20260925 01:32:30.484978 9432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299950, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:30.488611 9432 init.cc:377] Logged in from keytab as kudu/127.27.160.60@KRBTEST.COM (short username kudu)
I20260925 01:32:30.490300 9432 webserver.cc:460] Webserver started at http://127.27.160.60:46563/ using document root <none> and password file <none>
I20260925 01:32:30.491058 9432 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:30.491214 9432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:30.491541 9432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:32:30.494776 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/data/instance:
uuid: "b5e4574cd43e4ae2a9bae446bfd06527"
format_stamp: "Formatted at 2026-09-25 01:32:30 on dist-test-slave-h6hf"
server_key: "03a5a00c7e0023676d8febd11197d58d"
server_key_iv: "7e349b87c2d17b58504b32ce99fd54e9"
server_key_version: "encryptionkey@0"
I20260925 01:32:30.495625 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/wal/instance:
uuid: "b5e4574cd43e4ae2a9bae446bfd06527"
format_stamp: "Formatted at 2026-09-25 01:32:30 on dist-test-slave-h6hf"
server_key: "03a5a00c7e0023676d8febd11197d58d"
server_key_iv: "7e349b87c2d17b58504b32ce99fd54e9"
server_key_version: "encryptionkey@0"
I20260925 01:32:30.501981 9432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 01:32:30.506682 9447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:30.508301 9432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20260925 01:32:30.508556 9432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/wal
uuid: "b5e4574cd43e4ae2a9bae446bfd06527"
format_stamp: "Formatted at 2026-09-25 01:32:30 on dist-test-slave-h6hf"
server_key: "03a5a00c7e0023676d8febd11197d58d"
server_key_iv: "7e349b87c2d17b58504b32ce99fd54e9"
server_key_version: "encryptionkey@0"
I20260925 01:32:30.508809 9432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:30.520372 9432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:30.524895 9432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:30.525288 9432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:30.545092 9432 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.60:43403
I20260925 01:32:30.545158 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.60:43403 every 8 connection(s)
I20260925 01:32:30.546607 9432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
I20260925 01:32:30.550524 9499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:30.553975 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 9432
I20260925 01:32:30.554184 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790299737515179-28288-0/minicluster-data/master-2/wal/instance
I20260925 01:32:30.554749 28288 external_mini_cluster.cc:1428] Setting key 298f8a26542a094d47a5c1fb3bbdffa7
I20260925 01:32:30.558961 9499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299950, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:30.579393 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.562945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.27.160.62:37097)
0925 01:32:30.563145 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.563151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.563586 (+ 435us) server_negotiation.cc:407] Connection header received
0925 01:32:30.563834 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.563840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.563953 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.564103 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.565475 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.566230 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.567889 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.568578 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.568686 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.573459 (+ 4773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.573464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.573468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.573505 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.576123 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.576919 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.576926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.576930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.577089 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.577594 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.577599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.577603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.577894 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.578053 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.578716 (+ 663us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.578999 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:32:30.580581 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.557014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.27.160.60:43403)
0925 01:32:30.557932 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.557945 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.557994 (+ 49us) server_negotiation.cc:407] Connection header received
0925 01:32:30.559672 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.559697 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.560182 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.560630 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.562613 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.564823 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.566474 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.567305 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.567456 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.572968 (+ 5512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.572980 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.573001 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.573053 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.576237 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.577067 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.577076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.577080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.577328 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.577764 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.577769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.577772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.578352 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.578704 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.579055 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.579350 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":685,"spinlock_wait_cycles":2688,"thread_start_us":401,"threads_started":1}
I20260925 01:32:30.580621 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.562728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37097 (local address 127.0.0.1:47794)
0925 01:32:30.563443 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.563480 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:30.563687 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:30.564155 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:30.564163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:30.564843 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:30.565288 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.565302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.566399 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.566407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.567695 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.567712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.568849 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.568860 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.569051 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.569803 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:30.569849 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:30.573209 (+ 3360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:30.576287 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.576302 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.576320 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.576777 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.577264 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.577269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.577273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.577470 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.578026 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:30.578035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:30.578351 (+ 316us) client_negotiation.cc:769] Sending connection context
0925 01:32:30.578639 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 01:32:30.579218 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":46,"spinlock_wait_cycles":42112,"thread_start_us":412,"threads_started":1}
I20260925 01:32:30.583305 9499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299950, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:30.599192 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.584170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.27.160.61:34293)
0925 01:32:30.584403 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.584412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.584698 (+ 286us) server_negotiation.cc:407] Connection header received
0925 01:32:30.584869 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.584878 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.584969 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.585077 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.586293 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.587574 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.588858 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.589634 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.589784 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.593584 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.593590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.593593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.593630 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.596446 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.597155 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.597161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.597165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.597314 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.597755 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.597759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.597761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.598080 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.598231 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.598671 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.598849 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:32:30.599263 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.584091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:34293 (local address 127.0.0.1:48380)
0925 01:32:30.584626 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.584640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:30.584751 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:30.585100 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:30.585103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:30.585612 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:30.585946 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.585963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.587749 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.587755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.588689 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.588702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.589804 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.589810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.590044 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.590767 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:30.590792 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:30.593365 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:30.596599 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.596605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.596609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.597003 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.597433 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.597439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.597443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.597645 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.598202 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:30.598207 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:30.598407 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 01:32:30.598592 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 01:32:30.598910 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441}
I20260925 01:32:30.600666 9499 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:32:30.605669 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.586407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.27.160.62:37097)
0925 01:32:30.586631 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.586638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.589540 (+ 2902us) server_negotiation.cc:407] Connection header received
0925 01:32:30.589824 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.589832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.589929 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.590009 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.591944 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.592705 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.594173 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.594633 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.594792 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.599337 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.599347 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.599353 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.599422 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.602741 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.603494 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.603500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.603504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.603654 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.604162 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.604165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.604167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.604470 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.604684 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.605041 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.605308 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299950, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
I20260925 01:32:30.615674 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.601585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54856)
0925 01:32:30.601793 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.601811 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:30.601899 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:30.602191 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:30.602194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:30.602563 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:30.602697 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.602705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.603828 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.603834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.605072 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.605087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.606044 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.606049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.606234 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.606904 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:30.606925 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:30.609492 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:30.613071 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.613078 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.613082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.613546 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.613950 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.613956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.613964 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.614128 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.614901 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:30.614905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:30.615044 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 01:32:30.615160 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 01:32:30.615397 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 01:32:30.616166 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.601530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.27.160.60:43403)
0925 01:32:30.601737 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.601746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.601877 (+ 131us) server_negotiation.cc:407] Connection header received
0925 01:32:30.601983 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.601988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.602064 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.602168 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.602838 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.603697 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.605239 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.605863 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.606045 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.609692 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.609698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.609703 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.609746 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.612944 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.613645 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.613653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.613656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.613784 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.614333 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.614338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.614341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.614758 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.614939 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.615328 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.615613 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":69}
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:32:30.622471 9499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: Bootstrap starting.
I20260925 01:32:30.625209 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.607591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.27.160.61:34293)
0925 01:32:30.607774 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.607781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.607802 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:32:30.607896 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.607904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.607992 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.608102 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.609668 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.610562 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.612525 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.613027 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.613249 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.618674 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.618683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.618688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.618733 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.622166 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.623018 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.623027 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.623030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.623203 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.623658 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.623662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.623665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.624034 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.624320 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.624551 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.624814 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:32:30.626659 9499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:30.627913 9499 log.cc:824] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:30.630368 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:32:30.631165 9499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: No bootstrap required, opened a new log
I20260925 01:32:30.641873 9499 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.642549 9499 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:30.642731 9499 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e4574cd43e4ae2a9bae446bfd06527, State: Initialized, Role: FOLLOWER
I20260925 01:32:30.643682 9499 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.645530 9512 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.645949 9512 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:32:30.648030 9499 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:32:30.649796 9351 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 4
I20260925 01:32:30.657805 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.635490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.27.160.60:43403)
0925 01:32:30.635678 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.635685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.636408 (+ 723us) server_negotiation.cc:407] Connection header received
0925 01:32:30.636708 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.636714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.636798 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.636877 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.640151 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.641669 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.644453 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.645072 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.645222 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.649095 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.649102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.649106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.649179 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.653360 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.654381 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.654399 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.654402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.654598 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.655218 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.655226 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.655232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.655828 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.656128 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.656901 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.657223 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:32:30.665290 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.635262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.27.160.62:37097)
0925 01:32:30.635476 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.635483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.636665 (+ 1182us) server_negotiation.cc:407] Connection header received
0925 01:32:30.636923 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.636931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.637014 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.637085 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.639694 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.640826 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.643167 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.643814 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.643923 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.648715 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.648723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.648729 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.648790 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.654421 (+ 5631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.661543 (+ 7122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.661549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.661553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.661747 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.662591 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.662598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.662603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.663024 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.663305 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.664552 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.664830 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
I20260925 01:32:30.671069 9430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.636753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.27.160.61:34293)
0925 01:32:30.636937 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.636942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.645013 (+ 8071us) server_negotiation.cc:407] Connection header received
0925 01:32:30.645378 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.645387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.645476 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.645558 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.648036 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.649173 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.651903 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.652838 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.652971 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.657567 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.657577 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.657585 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.657644 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.662424 (+ 4780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.663590 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.663599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.663606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.663821 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.664438 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.664446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.664452 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.664942 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.668681 (+ 3739us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.670136 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.670505 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:32:30.682188 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.650454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54874)
0925 01:32:30.650854 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.650875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:30.650997 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:30.659028 (+ 8031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:30.659036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:30.659691 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:30.659976 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.659988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.661782 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.661790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.663375 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.663400 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.664638 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.664651 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.664845 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.665632 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:30.665667 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:30.669735 (+ 4068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:30.678991 (+ 9256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.678996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.679000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.679554 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.680008 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.680012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.680016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.680173 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.680890 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:30.680894 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:30.681066 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 01:32:30.681288 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 01:32:30.681810 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256}
I20260925 01:32:30.683307 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.651361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.27.160.60:43403)
0925 01:32:30.658577 (+ 7216us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.658585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.658608 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:32:30.658671 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.658678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.658800 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.660136 (+ 1336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.660285 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.661624 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.663653 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.664477 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.665180 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.671826 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.671836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.671841 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.671893 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.678819 (+ 6926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.679728 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.679731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.679734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.679876 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.680299 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.680304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.680306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.680752 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.680913 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.682530 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.682893 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7073,"thread_start_us":19436,"threads_started":1}
I20260925 01:32:30.689498 9351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b: Bootstrap starting.
I20260925 01:32:30.678516 9432 master_runner.cc:186] Error getting master registration for 127.27.160.62:37097: OK, instance_id { permanent_uuid: "74b980fa101a447197fb77f2951d552b" instance_seqno: 1790299949493697 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:32:30.690685 9432 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 01:32:30.690775 9524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:32:30.691787 9524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:32:30.693816 9351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:30.694881 9351 log.cc:824] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:30.697782 9351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b: No bootstrap required, opened a new log
I20260925 01:32:30.706210 9351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.706609 9351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:30.706748 9351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b980fa101a447197fb77f2951d552b, State: Initialized, Role: FOLLOWER
I20260925 01:32:30.707412 9351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.708475 9527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.708909 9527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:32:30.709909 9351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:32:30.723029 9538 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:32:30.723278 9538 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 01:32:30.733000 9305 tablet.cc:2369] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:32:30.720305 9284 master_runner.cc:186] Error getting master registration for 127.27.160.61:34293: OK, instance_id { permanent_uuid: "777ebf47a30044308023282911789c95" instance_seqno: 1790299950023177 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:32:30.733880 9284 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:32:30.753931 9425 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } attempt: 5
Sep 25 01:32:30 dist-test-slave-h6hf krb5kdc[9271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
I20260925 01:32:30.773102 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.754608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:43403 (local address 127.0.0.1:54878)
0925 01:32:30.754926 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:30.754948 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:30.755083 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:30.755466 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:30.755471 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:30.756040 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:30.756262 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.756275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.757791 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.757798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.759233 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:30.759249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.760289 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.760296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:30.760535 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.761286 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:30.761333 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:30.765410 (+ 4077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:30.770006 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.770013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.770018 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.770610 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.771096 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:30.771104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:30.771110 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:30.771299 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:30.771984 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:30.771990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:30.772163 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 01:32:30.772332 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 01:32:30.772671 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260925 01:32:30.773183 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:30.754704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.27.160.60:43403)
0925 01:32:30.754910 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:30.754917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:30.755030 (+ 113us) server_negotiation.cc:407] Connection header received
0925 01:32:30.755214 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:30.755220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:30.755301 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:30.755407 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:30.756431 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.757614 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.759408 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:30.760117 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:30.760252 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:30.765653 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:30.765660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:30.765666 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:30.765709 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:30.769814 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.770765 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.770771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.770775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.770960 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:30.771433 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:30.771438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:30.771442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:30.771840 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:30.772042 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:30.772437 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 01:32:30.772720 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 01:32:30.777524 9453 tablet.cc:2369] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:32:30.782497 9425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95: Bootstrap starting.
I20260925 01:32:30.787027 9425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:30.788596 9425 log.cc:824] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:30.792448 9425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95: No bootstrap required, opened a new log
I20260925 01:32:30.797503 9527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:32:30.797832 9527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.799571 9527 leader_election.cc:288] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 777ebf47a30044308023282911789c95 (127.27.160.61:34293), b5e4574cd43e4ae2a9bae446bfd06527 (127.27.160.60:43403)
I20260925 01:32:30.801625 9474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74b980fa101a447197fb77f2951d552b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" is_pre_election: true
I20260925 01:32:30.802294 9474 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74b980fa101a447197fb77f2951d552b in term 0.
I20260925 01:32:30.803248 9302 leader_election.cc:302] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b980fa101a447197fb77f2951d552b, b5e4574cd43e4ae2a9bae446bfd06527; no voters:
I20260925 01:32:30.803858 9527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:32:30.804020 9527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:32:30.804118 9527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:30.806151 9425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.806595 9425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:30.806713 9425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777ebf47a30044308023282911789c95, State: Initialized, Role: FOLLOWER
I20260925 01:32:30.807518 9425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.807648 9527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.809854 9474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74b980fa101a447197fb77f2951d552b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e4574cd43e4ae2a9bae446bfd06527"
I20260925 01:32:30.810114 9474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:30.815539 9474 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b980fa101a447197fb77f2951d552b in term 1.
I20260925 01:32:30.817869 9302 leader_election.cc:302] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b980fa101a447197fb77f2951d552b, b5e4574cd43e4ae2a9bae446bfd06527; no voters:
I20260925 01:32:30.821970 9527 leader_election.cc:288] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [CANDIDATE]: Term 1 election: Requested vote from peers 777ebf47a30044308023282911789c95 (127.27.160.61:34293), b5e4574cd43e4ae2a9bae446bfd06527 (127.27.160.60:43403)
I20260925 01:32:30.822139 9399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74b980fa101a447197fb77f2951d552b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777ebf47a30044308023282911789c95"
I20260925 01:32:30.822652 9527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:32:30.822666 9399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:30.823030 9527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [term 1 LEADER]: Becoming Leader. State: Replica: 74b980fa101a447197fb77f2951d552b, State: Running, Role: LEADER
I20260925 01:32:30.824421 9527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } }
I20260925 01:32:30.826241 9399 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74b980fa101a447197fb77f2951d552b in term 1.
I20260925 01:32:30.826649 9400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "74b980fa101a447197fb77f2951d552b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777ebf47a30044308023282911789c95" is_pre_election: true
I20260925 01:32:30.826998 9400 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 74b980fa101a447197fb77f2951d552b in term 1. Re-sending same reply.
I20260925 01:32:30.827296 9425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:32:30.828948 9541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.829907 9541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:32:30.840271 9542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b980fa101a447197fb77f2951d552b. Latest consensus state: current_term: 1 leader_uuid: "74b980fa101a447197fb77f2951d552b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.840859 9542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:30.860080 9546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:32:30.867659 9546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 01:32:30.876535 9555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:32:30.876819 9555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:32:30.887724 9474 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [term 1 FOLLOWER]: Refusing update from remote peer 74b980fa101a447197fb77f2951d552b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:32:30.890981 9542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:32:30.898715 9400 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [term 1 FOLLOWER]: Refusing update from remote peer 74b980fa101a447197fb77f2951d552b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:32:30.899817 9542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [LEADER]: Connected to new peer: Peer: permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:32:30.918723 9512 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b980fa101a447197fb77f2951d552b. Latest consensus state: current_term: 1 leader_uuid: "74b980fa101a447197fb77f2951d552b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.919046 9512 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:32:30.933694 9527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74b980fa101a447197fb77f2951d552b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.934092 9527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:30.934665 9557 mvcc.cc:205] Tried to move back new op lower bound from 7333068598822400000 to 7333068598592688128. Current Snapshot: MvccSnapshot[applied={T|T < 7333068598822400000}]
I20260925 01:32:30.937594 9541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74b980fa101a447197fb77f2951d552b. Latest consensus state: current_term: 1 leader_uuid: "74b980fa101a447197fb77f2951d552b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.937922 9541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:32:30.939193 9546 catalog_manager.cc:1347] Generated new cluster ID: f0ea69c22cac45a59d44fec007fe1aa2
I20260925 01:32:30.939317 9546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:32:30.939381 9512 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74b980fa101a447197fb77f2951d552b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.939646 9512 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:32:30.947273 9559 mvcc.cc:205] Tried to move back new op lower bound from 7333068598822400000 to 7333068598592688128. Current Snapshot: MvccSnapshot[applied={T|T < 7333068598822400000}]
I20260925 01:32:30.954411 9542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "74b980fa101a447197fb77f2951d552b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.955195 9541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "74b980fa101a447197fb77f2951d552b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b980fa101a447197fb77f2951d552b" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37097 } } peers { permanent_uuid: "777ebf47a30044308023282911789c95" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 34293 } } peers { permanent_uuid: "b5e4574cd43e4ae2a9bae446bfd06527" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 43403 } } }
I20260925 01:32:30.955515 9541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:32:30.956363 9542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:30.982559 9546 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:32:30.984673 9546 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:32:31.005715 9546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74b980fa101a447197fb77f2951d552b: Generated new TSK 0
I20260925 01:32:31.007166 9546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:32:31.694041 9524 catalog_manager.cc:1259] Loaded cluster ID: f0ea69c22cac45a59d44fec007fe1aa2
I20260925 01:32:31.694159 9524 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: loading cluster ID for follower catalog manager: success
I20260925 01:32:31.696674 9524 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: acquiring CA information for follower catalog manager: success
I20260925 01:32:31.698580 9524 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5e4574cd43e4ae2a9bae446bfd06527: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:32:31.878897 9555 catalog_manager.cc:1259] Loaded cluster ID: f0ea69c22cac45a59d44fec007fe1aa2
I20260925 01:32:31.879040 9555 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95: loading cluster ID for follower catalog manager: success
I20260925 01:32:31.881790 9555 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95: acquiring CA information for follower catalog manager: success
I20260925 01:32:31.883677 9555 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 777ebf47a30044308023282911789c95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:32:32.172230 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 9284
I20260925 01:32:32.173660 9346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:32.287029 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 9284
I20260925 01:32:32.311030 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 9358
I20260925 01:32:32.312211 9420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:32.423949 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 9358
I20260925 01:32:32.444439 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 9432
I20260925 01:32:32.445705 9494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:32.555439 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 9432
2026-09-25T01:32:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4891 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:32:32 dist-test-slave-h6hf krb5kdc[9575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:32:32 dist-test-slave-h6hf krb5kdc[9575](info): set up 2 sockets
Sep 25 01:32:32 dist-test-slave-h6hf krb5kdc[9575](info): commencing operation
krb5kdc: starting...
W20260925 01:32:33.691288 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:32:33 dist-test-slave-h6hf krb5kdc[9575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:32:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:32:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:32:33.954039 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:39285
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:39285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:34.300673 9588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:34.300998 9588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:34.301087 9588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:34.308637 9588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:34.308768 9588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:34.308815 9588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:34.308874 9588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:34.308921 9588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:34.319434 9588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:39285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:39285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:34.320799 9588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:34.322084 9588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:34.334249 9594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:34.334292 9596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:34.334208 9593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:34.335453 9588 server_base.cc:1034] running on GCE node
I20260925 01:32:34.336297 9588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:34.338042 9588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:34.339259 9588 hybrid_clock.cc:648] HybridClock initialized: now 1790299954339208 us; error 41 us; skew 500 ppm
I20260925 01:32:34.339639 9588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:34 dist-test-slave-h6hf krb5kdc[9575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299954, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:34.343011 9588 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:32:34.344590 9588 webserver.cc:460] Webserver started at http://127.27.160.62:41741/ using document root <none> and password file <none>
I20260925 01:32:34.345371 9588 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:34.345482 9588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:34.345746 9588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:32:34.348989 9588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "273c70d3bdec4c39bc19de2e2bd656cb"
format_stamp: "Formatted at 2026-09-25 01:32:34 on dist-test-slave-h6hf"
server_key: "68e2d1d4f368f655999d4abacb6181b0"
server_key_iv: "f69a0a37d30ae414d1cf8ee3a5f3d051"
server_key_version: "encryptionkey@0"
I20260925 01:32:34.349841 9588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "273c70d3bdec4c39bc19de2e2bd656cb"
format_stamp: "Formatted at 2026-09-25 01:32:34 on dist-test-slave-h6hf"
server_key: "68e2d1d4f368f655999d4abacb6181b0"
server_key_iv: "f69a0a37d30ae414d1cf8ee3a5f3d051"
server_key_version: "encryptionkey@0"
I20260925 01:32:34.355230 9588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:32:34.358880 9603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:34.360339 9588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:32:34.360538 9588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "273c70d3bdec4c39bc19de2e2bd656cb"
format_stamp: "Formatted at 2026-09-25 01:32:34 on dist-test-slave-h6hf"
server_key: "68e2d1d4f368f655999d4abacb6181b0"
server_key_iv: "f69a0a37d30ae414d1cf8ee3a5f3d051"
server_key_version: "encryptionkey@0"
I20260925 01:32:34.360807 9588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:34.386883 9588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:34.391522 9588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:34.391826 9588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:34.411465 9588 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:39285
I20260925 01:32:34.411504 9654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:39285 every 8 connection(s)
I20260925 01:32:34.413046 9588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:32:34.417086 9655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:34.419238 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 9588
I20260925 01:32:34.419503 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:32:34.420360 28288 external_mini_cluster.cc:1428] Setting key 42c8fbfed942dc7fb3b76090e14bab9a
I20260925 01:32:34.429998 9655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb: Bootstrap starting.
I20260925 01:32:34.433658 9655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:34.434751 9655 log.cc:824] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:32:34 dist-test-slave-h6hf krb5kdc[9575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:34.438089 9655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb: No bootstrap required, opened a new log
I20260925 01:32:34.447017 9655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273c70d3bdec4c39bc19de2e2bd656cb" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39285 } }
I20260925 01:32:34.447574 9655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:34.447706 9655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273c70d3bdec4c39bc19de2e2bd656cb, State: Initialized, Role: FOLLOWER
I20260925 01:32:34.448330 9655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273c70d3bdec4c39bc19de2e2bd656cb" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39285 } }
I20260925 01:32:34.448611 9655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:32:34.448659 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:34.422622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.27.160.62:39285)
0925 01:32:34.423475 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:34.423492 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:34.423542 (+ 50us) server_negotiation.cc:407] Connection header received
0925 01:32:34.425366 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:34.425392 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:34.425833 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:34.426253 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:34.427956 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:34.429409 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:34.430912 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:34.431644 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:34.431867 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:34.436986 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:34.436995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:34.437011 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:34.437067 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:34.443758 (+ 6691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:34.444657 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:34.444666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:34.444673 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:34.444856 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:34.445376 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:34.445392 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:34.445394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:34.446006 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:34.446350 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:34.446820 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 01:32:34.447304 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":342,"threads_started":1}
I20260925 01:32:34.448781 9655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:32:34.448992 9655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:34.451943 9655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273c70d3bdec4c39bc19de2e2bd656cb" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39285 } }
I20260925 01:32:34.452492 9655 leader_election.cc:302] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273c70d3bdec4c39bc19de2e2bd656cb; no voters:
I20260925 01:32:34.453169 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:32:34.453248 9655 leader_election.cc:288] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:32:34.453369 9660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:32:34.455425 9655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:32:34.460202 9660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [term 1 LEADER]: Becoming Leader. State: Replica: 273c70d3bdec4c39bc19de2e2bd656cb, State: Running, Role: LEADER
I20260925 01:32:34.460774 9660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273c70d3bdec4c39bc19de2e2bd656cb" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39285 } }
I20260925 01:32:34.464269 9662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273c70d3bdec4c39bc19de2e2bd656cb. Latest consensus state: current_term: 1 leader_uuid: "273c70d3bdec4c39bc19de2e2bd656cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273c70d3bdec4c39bc19de2e2bd656cb" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39285 } } }
I20260925 01:32:34.464259 9661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273c70d3bdec4c39bc19de2e2bd656cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273c70d3bdec4c39bc19de2e2bd656cb" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39285 } } }
I20260925 01:32:34.464705 9662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:34.464730 9661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:34.466481 9670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:32:34.470948 9670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:32:34.479460 9670 catalog_manager.cc:1347] Generated new cluster ID: 5175c79a2bd7495c806949650806cd6a
I20260925 01:32:34.479595 9670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:32:34.492071 9670 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:32:34.493528 9670 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:32:34.505228 9670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273c70d3bdec4c39bc19de2e2bd656cb: Generated new TSK 0
I20260925 01:32:34.506208 9670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:32:34.516963 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 9588
I20260925 01:32:34.518472 9650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:34.611511 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 9588
2026-09-25T01:32:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2052 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:32:34 dist-test-slave-h6hf krb5kdc[9683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:32:34 dist-test-slave-h6hf krb5kdc[9683](info): set up 2 sockets
Sep 25 01:32:34 dist-test-slave-h6hf krb5kdc[9683](info): commencing operation
krb5kdc: starting...
W20260925 01:32:35.742834 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:32:35 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:32:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:32:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:32:35.996480 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40651
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:40651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:36.338964 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:36.339273 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:36.339450 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:36.346843 9696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:36.346977 9696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:36.347075 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:36.347178 9696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:36.347246 9696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:36.357689 9696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:40651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:36.359158 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:36.360532 9696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:36.373330 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:36.373330 9702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:36.373431 9704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:36.374182 9696 server_base.cc:1034] running on GCE node
I20260925 01:32:36.375118 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:36.376734 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:36.377975 9696 hybrid_clock.cc:648] HybridClock initialized: now 1790299956377916 us; error 54 us; skew 500 ppm
I20260925 01:32:36.378346 9696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:36 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:36.381757 9696 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:32:36.383374 9696 webserver.cc:460] Webserver started at http://127.27.160.62:39581/ using document root <none> and password file <none>
I20260925 01:32:36.384106 9696 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:36.384258 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:36.384574 9696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:32:36.387789 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "2387cfabf2d84795915ecb496461ee14"
format_stamp: "Formatted at 2026-09-25 01:32:36 on dist-test-slave-h6hf"
server_key: "fddf8d78b5a8f5cca68544f02215195e"
server_key_iv: "603885e2c5b98993f4852961065c7f66"
server_key_version: "encryptionkey@0"
I20260925 01:32:36.388641 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "2387cfabf2d84795915ecb496461ee14"
format_stamp: "Formatted at 2026-09-25 01:32:36 on dist-test-slave-h6hf"
server_key: "fddf8d78b5a8f5cca68544f02215195e"
server_key_iv: "603885e2c5b98993f4852961065c7f66"
server_key_version: "encryptionkey@0"
I20260925 01:32:36.394423 9696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 01:32:36.398309 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:36.399847 9696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260925 01:32:36.400049 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "2387cfabf2d84795915ecb496461ee14"
format_stamp: "Formatted at 2026-09-25 01:32:36 on dist-test-slave-h6hf"
server_key: "fddf8d78b5a8f5cca68544f02215195e"
server_key_iv: "603885e2c5b98993f4852961065c7f66"
server_key_version: "encryptionkey@0"
I20260925 01:32:36.400271 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:36.433498 9696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:36.438277 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:36.438658 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:36.459475 9696 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:40651
I20260925 01:32:36.459532 9762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:40651 every 8 connection(s)
I20260925 01:32:36.461297 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:32:36.462954 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 9696
I20260925 01:32:36.463225 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:32:36.463996 28288 external_mini_cluster.cc:1428] Setting key d7f5a7529f82dfe68caf6eda083f3374
I20260925 01:32:36.466091 9763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:32:36 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:36.482507 9763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Bootstrap starting.
I20260925 01:32:36.487818 9763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:36.489575 9763 log.cc:824] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:36.492457 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:36.466505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.27.160.62:40651)
0925 01:32:36.467382 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:36.467399 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:36.467455 (+ 56us) server_negotiation.cc:407] Connection header received
0925 01:32:36.469738 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:36.469773 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:36.470476 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:36.471020 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:36.472529 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:36.475062 (+ 2533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:36.476471 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:36.477709 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:36.477948 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:36.482534 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:36.482551 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:36.482575 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:36.482641 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:36.487280 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:36.488180 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:36.488192 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:36.488201 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:36.488500 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:36.488914 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:36.488921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:36.488926 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:36.489924 (+ 998us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:36.490332 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:36.490792 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 01:32:36.491304 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":359,"threads_started":1}
I20260925 01:32:36.493101 9763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: No bootstrap required, opened a new log
I20260925 01:32:36.501605 9763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:36.502051 9763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:36.502168 9763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2387cfabf2d84795915ecb496461ee14, State: Initialized, Role: FOLLOWER
I20260925 01:32:36.502770 9763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:36.503028 9763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:32:36.503175 9763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:32:36.503369 9763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:36.505947 9763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:36.506564 9763 leader_election.cc:302] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2387cfabf2d84795915ecb496461ee14; no voters:
I20260925 01:32:36.507269 9763 leader_election.cc:288] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:32:36.507447 9768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:32:36.508518 9768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 1 LEADER]: Becoming Leader. State: Replica: 2387cfabf2d84795915ecb496461ee14, State: Running, Role: LEADER
I20260925 01:32:36.509078 9768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:36.509413 9763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:32:36.512179 9770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2387cfabf2d84795915ecb496461ee14. Latest consensus state: current_term: 1 leader_uuid: "2387cfabf2d84795915ecb496461ee14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } } }
I20260925 01:32:36.512213 9769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2387cfabf2d84795915ecb496461ee14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } } }
I20260925 01:32:36.512639 9769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:36.512639 9770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:36.515146 9776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:32:36.520128 9776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:32:36.530253 9776 catalog_manager.cc:1347] Generated new cluster ID: 286a907ab3954d24bf765f3be254655e
I20260925 01:32:36.530448 9776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:32:36.540102 9776 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:32:36.542323 9776 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:32:36.552544 9776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Generated new TSK 0
I20260925 01:32:36.553704 9776 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:32:36.621544 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40651
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:36.973706 9789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:36.974066 9789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:36.974246 9789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:36.981753 9789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:36.981931 9789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:36.982127 9789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:32:36.993060 9789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:36.994921 9789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:36.996858 9789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:37.011483 9794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:37.011559 9795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:37.011520 9797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:37.013159 9789 server_base.cc:1034] running on GCE node
I20260925 01:32:37.013926 9789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:37.015143 9789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:37.016386 9789 hybrid_clock.cc:648] HybridClock initialized: now 1790299957016312 us; error 70 us; skew 500 ppm
I20260925 01:32:37.016733 9789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:37 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:37.020256 9789 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:32:37.022084 9789 webserver.cc:460] Webserver started at http://127.27.160.1:44945/ using document root <none> and password file <none>
I20260925 01:32:37.022851 9789 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:37.023010 9789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:37.023343 9789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:32:37.026510 9789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "faed72eb6aff47da88f59700c3c3666b"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1f0605ff5d6262d29c6ee0250c74207e"
server_key_iv: "33c25efc564ce326605587391e8cbbf8"
server_key_version: "encryptionkey@0"
I20260925 01:32:37.027362 9789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "faed72eb6aff47da88f59700c3c3666b"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1f0605ff5d6262d29c6ee0250c74207e"
server_key_iv: "33c25efc564ce326605587391e8cbbf8"
server_key_version: "encryptionkey@0"
I20260925 01:32:37.032832 9789 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:32:37.036644 9804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:37.038156 9789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:32:37.038339 9789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "faed72eb6aff47da88f59700c3c3666b"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1f0605ff5d6262d29c6ee0250c74207e"
server_key_iv: "33c25efc564ce326605587391e8cbbf8"
server_key_version: "encryptionkey@0"
I20260925 01:32:37.038566 9789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:37.067319 9789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:37.071847 9789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:37.072178 9789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:37.073274 9789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:37.074759 9789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:37.074867 9789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:37.075034 9789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:37.075115 9789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:37.111096 9789 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:33579
I20260925 01:32:37.111157 9916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:33579 every 8 connection(s)
I20260925 01:32:37.112462 9789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:32:37.120976 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 9789
I20260925 01:32:37.121258 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:32:37.121917 28288 external_mini_cluster.cc:1428] Setting key 352c2fd5774848f8b644ca0f265e0a54
Sep 25 01:32:37 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:37.136415 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:37.115379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:46143 (local address 127.27.160.62:40651)
0925 01:32:37.115808 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:37.115813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:37.116956 (+ 1143us) server_negotiation.cc:407] Connection header received
0925 01:32:37.118961 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:37.118967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:37.119060 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:37.119189 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:37.120859 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:37.121733 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:37.123911 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:37.124498 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:37.124655 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:37.128653 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:37.128664 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:37.128673 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:37.128724 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:37.132571 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:37.133853 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:37.133858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:37.133860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:37.133991 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:37.134529 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:37.134532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:37.134534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:37.134808 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:37.134923 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:37.135934 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0925 01:32:37.136116 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":204,"threads_started":1}
I20260925 01:32:37.138109 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:37.116019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:46143)
0925 01:32:37.116774 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:37.116812 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:37.118646 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:37.119447 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:37.119455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:37.120013 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:37.120656 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:37.120676 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:37.121921 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:37.121925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:37.123715 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:37.123730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:37.124735 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:37.124817 (+ 82us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:37.125036 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:37.125919 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:37.125950 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:37.128347 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:37.132851 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:37.132864 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:37.132888 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:37.133700 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:37.134112 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:37.134119 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:37.134126 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:37.134402 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:37.134938 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:37.134950 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:37.135510 (+ 560us) client_negotiation.cc:769] Sending connection context
0925 01:32:37.135842 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 01:32:37.136496 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":314,"threads_started":1}
I20260925 01:32:37.139954 9917 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40651
I20260925 01:32:37.140403 9917 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:37.141377 9917 heartbeater.cc:511] Master 127.27.160.62:40651 requested a full tablet report, sending...
I20260925 01:32:37.144305 9728 ts_manager.cc:194] Registered new tserver with Master: faed72eb6aff47da88f59700c3c3666b (127.27.160.1:33579)
I20260925 01:32:37.146790 9728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:46143
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:32:37.181435 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40651
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:37.531605 9923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:37.532130 9923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:37.532339 9923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:37.542749 9923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:37.542927 9923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:37.543125 9923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:32:37.553642 9923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:37.555091 9923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:37.556504 9923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:37.570822 9931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:37.570832 9929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:37.570943 9928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:37.571528 9923 server_base.cc:1034] running on GCE node
I20260925 01:32:37.572358 9923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:37.573426 9923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:37.574638 9923 hybrid_clock.cc:648] HybridClock initialized: now 1790299957574587 us; error 39 us; skew 500 ppm
I20260925 01:32:37.575026 9923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:37 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:37.578675 9923 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:32:37.580246 9923 webserver.cc:460] Webserver started at http://127.27.160.2:39567/ using document root <none> and password file <none>
I20260925 01:32:37.581038 9923 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:37.581223 9923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:37.581553 9923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:32:37.584611 9923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1feece858f52b0609bdd2b37cd9a2ce3"
server_key_iv: "736a2052663b539dbe3eb6cc97e145f2"
server_key_version: "encryptionkey@0"
I20260925 01:32:37.585598 9923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1feece858f52b0609bdd2b37cd9a2ce3"
server_key_iv: "736a2052663b539dbe3eb6cc97e145f2"
server_key_version: "encryptionkey@0"
I20260925 01:32:37.591214 9923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:32:37.595127 9938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:37.596787 9923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:32:37.596992 9923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1feece858f52b0609bdd2b37cd9a2ce3"
server_key_iv: "736a2052663b539dbe3eb6cc97e145f2"
server_key_version: "encryptionkey@0"
I20260925 01:32:37.597311 9923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:37.619987 9923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:37.625422 9923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:37.625784 9923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:37.626906 9923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:37.630532 9923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:37.630664 9923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:37.630836 9923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:37.630911 9923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:37.669303 9923 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:46021
I20260925 01:32:37.669358 10050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:46021 every 8 connection(s)
I20260925 01:32:37.670734 9923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:32:37.676973 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 9923
I20260925 01:32:37.677237 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:32:37.677874 28288 external_mini_cluster.cc:1428] Setting key 35c4e4afa5789a4ab1f7011de7b006c9
Sep 25 01:32:37 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:37.697917 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:37.673594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:43697 (local address 127.27.160.62:40651)
0925 01:32:37.674052 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:37.674057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:37.675386 (+ 1329us) server_negotiation.cc:407] Connection header received
0925 01:32:37.678071 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:37.678077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:37.678161 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:37.678290 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:37.680510 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:37.681416 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:37.684463 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:37.684890 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:37.684995 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:37.690321 (+ 5326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:37.690331 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:37.690338 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:37.690393 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:37.694328 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:37.695147 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:37.695156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:37.695162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:37.695335 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:37.695763 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:37.695771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:37.695777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:37.696189 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:37.696359 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:37.697051 (+ 692us) server_negotiation.cc:299] Negotiation successful
0925 01:32:37.697400 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260925 01:32:37.698617 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:37.674257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:43697)
0925 01:32:37.675177 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:37.675228 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:37.677591 (+ 2363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:37.678635 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:37.678645 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:37.679426 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:37.680262 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:37.680293 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:37.681593 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:37.681603 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:37.684255 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:37.684279 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:37.685192 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:37.685204 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:37.685606 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:37.686497 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:37.686541 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:37.689994 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:37.694502 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:37.694511 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:37.694526 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:37.695007 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:37.695453 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:37.695456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:37.695459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:37.695651 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:37.696316 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:37.696322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:37.696665 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 01:32:37.696926 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 01:32:37.697439 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":417,"threads_started":1}
I20260925 01:32:37.700218 10051 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40651
I20260925 01:32:37.700587 10051 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:37.701623 10051 heartbeater.cc:511] Master 127.27.160.62:40651 requested a full tablet report, sending...
I20260925 01:32:37.703284 9728 ts_manager.cc:194] Registered new tserver with Master: 582e38dd0b584d5a8f4f8bf79ad00c1e (127.27.160.2:46021)
I20260925 01:32:37.704186 9728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:43697
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:32:37.737363 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40651
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:38.081615 10057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:38.081979 10057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:38.082165 10057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:38.089524 10057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:38.089701 10057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:38.089954 10057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:32:38.100041 10057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:38.101605 10057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:38.103120 10057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:38.118089 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:38.117954 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:38.117985 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:38.118564 10057 server_base.cc:1034] running on GCE node
I20260925 01:32:38.119313 10057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:38.120292 10057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:38.121521 10057 hybrid_clock.cc:648] HybridClock initialized: now 1790299958121456 us; error 43 us; skew 500 ppm
I20260925 01:32:38.121876 10057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:38 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299958, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:38.125216 10057 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:32:38.126787 10057 webserver.cc:460] Webserver started at http://127.27.160.3:42139/ using document root <none> and password file <none>
I20260925 01:32:38.127568 10057 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:38.127732 10057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:38.128065 10057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:32:38.131312 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "054e01f6aa2347c890e8f4e834006bef"
format_stamp: "Formatted at 2026-09-25 01:32:38 on dist-test-slave-h6hf"
server_key: "1709202b120096b14f5079b7eaeaeb80"
server_key_iv: "fe84fcffaf22cae1b5bf94f335e7a7b9"
server_key_version: "encryptionkey@0"
I20260925 01:32:38.132158 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "054e01f6aa2347c890e8f4e834006bef"
format_stamp: "Formatted at 2026-09-25 01:32:38 on dist-test-slave-h6hf"
server_key: "1709202b120096b14f5079b7eaeaeb80"
server_key_iv: "fe84fcffaf22cae1b5bf94f335e7a7b9"
server_key_version: "encryptionkey@0"
I20260925 01:32:38.137800 10057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:32:38.141822 10072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:38.143757 10057 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:32:38.143970 10057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "054e01f6aa2347c890e8f4e834006bef"
format_stamp: "Formatted at 2026-09-25 01:32:38 on dist-test-slave-h6hf"
server_key: "1709202b120096b14f5079b7eaeaeb80"
server_key_iv: "fe84fcffaf22cae1b5bf94f335e7a7b9"
server_key_version: "encryptionkey@0"
I20260925 01:32:38.144219 10057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:38.150446 9917 heartbeater.cc:503] Master 127.27.160.62:40651 was elected leader, sending a full tablet report...
I20260925 01:32:38.160928 10057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:38.165772 10057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:38.166095 10057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:38.167236 10057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:38.168785 10057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:38.168874 10057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:38.168987 10057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:38.169045 10057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:38.208415 10057 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:37535
I20260925 01:32:38.208461 10184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:37535 every 8 connection(s)
I20260925 01:32:38.210089 10057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:32:38.212337 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 10057
I20260925 01:32:38.212589 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:32:38.213114 28288 external_mini_cluster.cc:1428] Setting key 3d230a01382abc9b657a539dc0c0c1aa
Sep 25 01:32:38 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299958, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:38.232812 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.213340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:51911 (local address 127.27.160.62:40651)
0925 01:32:38.213983 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.213989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.215197 (+ 1208us) server_negotiation.cc:407] Connection header received
0925 01:32:38.217014 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.217018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.217092 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.217239 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:38.218900 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.219682 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.221515 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.221943 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.222044 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.225933 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:38.225940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:38.225943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:38.225980 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:38.229436 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.230332 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.230337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.230340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.230493 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.231021 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.231025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.231027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.231336 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:38.231473 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.232274 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.232468 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":316,"threads_started":1}
I20260925 01:32:38.233821 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.214062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:51911)
0925 01:32:38.215020 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:38.215065 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:38.216766 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:38.217476 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:38.217485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:38.218102 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:38.218724 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.218744 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.219845 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.219850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.221363 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.221377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.222171 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.222177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.222412 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.223088 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:38.223111 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:38.225607 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:38.229649 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:38.229658 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:38.229675 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:38.230183 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:38.230600 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:38.230603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:38.230605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:38.230796 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:38.231473 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:38.231479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:38.231854 (+ 375us) client_negotiation.cc:769] Sending connection context
0925 01:32:38.232107 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 01:32:38.232549 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":372,"threads_started":1}
I20260925 01:32:38.235622 10185 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40651
I20260925 01:32:38.235925 10185 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:38.236682 10185 heartbeater.cc:511] Master 127.27.160.62:40651 requested a full tablet report, sending...
I20260925 01:32:38.238420 9728 ts_manager.cc:194] Registered new tserver with Master: 054e01f6aa2347c890e8f4e834006bef (127.27.160.3:37535)
I20260925 01:32:38.239413 9728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:51911
I20260925 01:32:38.241812 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:32:38.261451 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.247266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.27.160.62:40651)
0925 01:32:38.247451 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.247456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.247951 (+ 495us) server_negotiation.cc:407] Connection header received
0925 01:32:38.248109 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.248118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.248193 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.248274 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:38.250396 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.251143 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.252663 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.253086 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.253259 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.255868 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:38.255874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:38.255877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:38.255944 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:38.258738 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.259515 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.259519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.259522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.259650 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.260069 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.260072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.260074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.260412 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:38.260598 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.260943 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.261198 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:32:38.266158 9727 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:32:38.273631 9727 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:32:38.301582 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.287198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:37535 (local address 127.0.0.1:53622)
0925 01:32:38.288360 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:38.288417 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:38.288668 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:38.290302 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:38.290311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:38.290345 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:38.290976 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.290992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.294165 (+ 3173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.294176 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.297461 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.297486 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.300424 (+ 2938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.300436 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.300702 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.300778 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 01:32:38.300914 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 01:32:38.301166 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":853,"threads_started":1}
I20260925 01:32:38.303027 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.287571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.27.160.3:37535)
0925 01:32:38.289499 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.289508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.289553 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:32:38.289673 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.289682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.289991 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.291234 (+ 1243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:38.291581 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.293910 (+ 2329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.297913 (+ 4003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.300041 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.300818 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.302319 (+ 1501us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.302414 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.302628 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1758,"thread_start_us":1564,"threads_started":1}
I20260925 01:32:38.307583 10120 tablet_service.cc:1467] Processing CreateTablet for tablet 82f1c6959c094577b6cb3bdc6b2b24a1 (DEFAULT_TABLE table=test-table [id=9b191d6030d84c6d976d1653f8b25da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:38.309444 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.287490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:46021 (local address 127.0.0.1:48042)
0925 01:32:38.299738 (+ 12248us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:38.299770 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:38.299979 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:38.301845 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:38.301852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:38.301887 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:38.302423 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.302439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.304525 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.304531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.306667 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.306689 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.308476 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.308483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.308667 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.308718 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 01:32:38.308829 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:32:38.309004 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12069,"thread_start_us":15052,"threads_started":1}
I20260925 01:32:38.309628 10120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f1c6959c094577b6cb3bdc6b2b24a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:38.310114 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.287765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.27.160.2:46021)
0925 01:32:38.300891 (+ 13126us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.300905 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.300952 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:32:38.301143 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.301170 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.301566 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.301903 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:38.302713 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.304356 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.306909 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.308225 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.308420 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.308661 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.308989 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.309241 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12854,"thread_start_us":12700,"threads_started":1}
I20260925 01:32:38.311491 10198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.290631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:33579 (local address 127.0.0.1:53730)
0925 01:32:38.301846 (+ 11215us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:38.301867 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:38.301974 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:38.303726 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:38.303734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:38.303795 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:38.304030 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.304043 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.306236 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.306242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.308199 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.308220 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.310630 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.310638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.310846 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.310887 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 01:32:38.310986 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 01:32:38.311196 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11115,"thread_start_us":9216,"threads_started":1}
I20260925 01:32:38.312031 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.291856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.27.160.1:33579)
0925 01:32:38.299805 (+ 7949us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.299817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.302846 (+ 3029us) server_negotiation.cc:407] Connection header received
0925 01:32:38.302972 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.302981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.303472 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.303703 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:38.304238 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.306031 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.308798 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.310414 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.310606 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.310884 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.311428 (+ 544us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.311648 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7751,"thread_start_us":7583,"threads_started":1}
I20260925 01:32:38.314932 9986 tablet_service.cc:1467] Processing CreateTablet for tablet 82f1c6959c094577b6cb3bdc6b2b24a1 (DEFAULT_TABLE table=test-table [id=9b191d6030d84c6d976d1653f8b25da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:38.316376 9986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f1c6959c094577b6cb3bdc6b2b24a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:38.316814 9852 tablet_service.cc:1467] Processing CreateTablet for tablet 82f1c6959c094577b6cb3bdc6b2b24a1 (DEFAULT_TABLE table=test-table [id=9b191d6030d84c6d976d1653f8b25da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:38.318377 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f1c6959c094577b6cb3bdc6b2b24a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:38.328658 10204 tablet_bootstrap.cc:492] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Bootstrap starting.
I20260925 01:32:38.329905 10205 tablet_bootstrap.cc:492] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Bootstrap starting.
I20260925 01:32:38.332592 10204 tablet_bootstrap.cc:654] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:38.333438 10205 tablet_bootstrap.cc:654] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:38.333845 10206 tablet_bootstrap.cc:492] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Bootstrap starting.
I20260925 01:32:38.334059 10204 log.cc:824] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:38.335263 10205 log.cc:824] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:38.337846 10204 tablet_bootstrap.cc:492] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: No bootstrap required, opened a new log
I20260925 01:32:38.337905 10206 tablet_bootstrap.cc:654] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:38.338265 10204 ts_tablet_manager.cc:1397] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 01:32:38.338958 10205 tablet_bootstrap.cc:492] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: No bootstrap required, opened a new log
I20260925 01:32:38.339277 10205 ts_tablet_manager.cc:1397] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260925 01:32:38.339377 10206 log.cc:824] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:38.343022 10206 tablet_bootstrap.cc:492] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: No bootstrap required, opened a new log
I20260925 01:32:38.343351 10206 ts_tablet_manager.cc:1397] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260925 01:32:38.347119 10204 raft_consensus.cc:348] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.347545 10204 raft_consensus.cc:374] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:38.347651 10204 raft_consensus.cc:729] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054e01f6aa2347c890e8f4e834006bef, State: Initialized, Role: FOLLOWER
I20260925 01:32:38.348847 10204 consensus_queue.cc:260] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.350283 10185 heartbeater.cc:503] Master 127.27.160.62:40651 was elected leader, sending a full tablet report...
I20260925 01:32:38.351233 10204 ts_tablet_manager.cc:1428] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 01:32:38.351091 10205 raft_consensus.cc:348] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.351601 10205 raft_consensus.cc:374] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:38.351724 10205 raft_consensus.cc:729] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582e38dd0b584d5a8f4f8bf79ad00c1e, State: Initialized, Role: FOLLOWER
I20260925 01:32:38.352420 10205 consensus_queue.cc:260] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.353796 10051 heartbeater.cc:503] Master 127.27.160.62:40651 was elected leader, sending a full tablet report...
I20260925 01:32:38.354509 10205 ts_tablet_manager.cc:1428] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260925 01:32:38.355991 10206 raft_consensus.cc:348] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.356503 10206 raft_consensus.cc:374] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:38.356613 10206 raft_consensus.cc:729] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faed72eb6aff47da88f59700c3c3666b, State: Initialized, Role: FOLLOWER
I20260925 01:32:38.357278 10206 consensus_queue.cc:260] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.359297 10206 ts_tablet_manager.cc:1428] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
W20260925 01:32:38.365339 9918 tablet.cc:2369] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:32:38.422994 10052 tablet.cc:2369] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:32:38.459470 10212 raft_consensus.cc:482] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:32:38.459806 10212 raft_consensus.cc:504] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.461927 10212 leader_election.cc:288] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 054e01f6aa2347c890e8f4e834006bef (127.27.160.3:37535), 582e38dd0b584d5a8f4f8bf79ad00c1e (127.27.160.2:46021)
W20260925 01:32:38.462054 10186 tablet.cc:2369] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:32:38.471614 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.462800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:59779 (local address 127.27.160.2:46021)
0925 01:32:38.463210 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.463219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.464371 (+ 1152us) server_negotiation.cc:407] Connection header received
0925 01:32:38.464593 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.464601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.464689 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.464758 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:38.465528 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.466232 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.469396 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.470669 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.470843 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.470919 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.471197 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.471335 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260925 01:32:38.471871 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.462448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:46021 (local address 127.27.160.1:59779)
0925 01:32:38.464261 (+ 1813us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:38.464285 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:38.464422 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:38.464972 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:38.464976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:38.465000 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:38.465362 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.465370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.466391 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.466398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.469193 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.469210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.470889 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.470893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.471022 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.471047 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:32:38.471164 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 01:32:38.471306 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1681,"mutex_wait_us":154,"thread_start_us":1473,"threads_started":1}
I20260925 01:32:38.473285 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.462260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:37535 (local address 127.27.160.1:43789)
0925 01:32:38.463832 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:38.463857 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:38.464122 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:38.464521 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:38.464526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:38.464562 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:38.464984 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.464995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.466428 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.466435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.469193 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:38.469210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.472478 (+ 3268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.472488 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:38.472688 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.472720 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:32:38.472824 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:32:38.472931 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"mutex_wait_us":119,"thread_start_us":1216,"threads_started":1}
I20260925 01:32:38.473325 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.462718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:43789 (local address 127.27.160.3:37535)
0925 01:32:38.463034 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.463045 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.463982 (+ 937us) server_negotiation.cc:407] Connection header received
0925 01:32:38.464303 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.464309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.464386 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.464490 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:38.465167 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.466280 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.471007 (+ 4727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.472263 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.472427 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.472497 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.472874 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.473024 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 01:32:38.473105 10006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f1c6959c094577b6cb3bdc6b2b24a1" candidate_uuid: "faed72eb6aff47da88f59700c3c3666b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" is_pre_election: true
I20260925 01:32:38.473773 10006 raft_consensus.cc:2454] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faed72eb6aff47da88f59700c3c3666b in term 0.
I20260925 01:32:38.474427 10140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f1c6959c094577b6cb3bdc6b2b24a1" candidate_uuid: "faed72eb6aff47da88f59700c3c3666b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "054e01f6aa2347c890e8f4e834006bef" is_pre_election: true
I20260925 01:32:38.474586 9808 leader_election.cc:302] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 582e38dd0b584d5a8f4f8bf79ad00c1e, faed72eb6aff47da88f59700c3c3666b; no voters:
I20260925 01:32:38.474987 10140 raft_consensus.cc:2454] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faed72eb6aff47da88f59700c3c3666b in term 0.
I20260925 01:32:38.475080 10212 raft_consensus.cc:2781] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:32:38.475237 10212 raft_consensus.cc:482] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:32:38.475400 10212 raft_consensus.cc:3037] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:38.477824 10212 raft_consensus.cc:504] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.478526 10212 leader_election.cc:288] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [CANDIDATE]: Term 1 election: Requested vote from peers 054e01f6aa2347c890e8f4e834006bef (127.27.160.3:37535), 582e38dd0b584d5a8f4f8bf79ad00c1e (127.27.160.2:46021)
I20260925 01:32:38.479043 10140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f1c6959c094577b6cb3bdc6b2b24a1" candidate_uuid: "faed72eb6aff47da88f59700c3c3666b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "054e01f6aa2347c890e8f4e834006bef"
I20260925 01:32:38.479077 10006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82f1c6959c094577b6cb3bdc6b2b24a1" candidate_uuid: "faed72eb6aff47da88f59700c3c3666b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e"
I20260925 01:32:38.479274 10140 raft_consensus.cc:3037] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:38.479301 10006 raft_consensus.cc:3037] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:38.481653 10006 raft_consensus.cc:2454] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faed72eb6aff47da88f59700c3c3666b in term 1.
I20260925 01:32:38.481654 10140 raft_consensus.cc:2454] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faed72eb6aff47da88f59700c3c3666b in term 1.
I20260925 01:32:38.482143 9806 leader_election.cc:302] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054e01f6aa2347c890e8f4e834006bef, faed72eb6aff47da88f59700c3c3666b; no voters:
I20260925 01:32:38.482555 10212 raft_consensus.cc:2781] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:32:38.483068 10212 raft_consensus.cc:686] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 1 LEADER]: Becoming Leader. State: Replica: faed72eb6aff47da88f59700c3c3666b, State: Running, Role: LEADER
I20260925 01:32:38.483611 10212 consensus_queue.cc:237] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } }
I20260925 01:32:38.488430 9727 catalog_manager.cc:5495] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b reported cstate change: term changed from 0 to 1, leader changed from <none> to faed72eb6aff47da88f59700c3c3666b (127.27.160.1). New cstate: current_term: 1 leader_uuid: "faed72eb6aff47da88f59700c3c3666b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } health_report { overall_health: HEALTHY } } }
I20260925 01:32:38.551924 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.545946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.27.160.1:33579)
0925 01:32:38.546210 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.546216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.546328 (+ 112us) server_negotiation.cc:407] Connection header received
0925 01:32:38.546604 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.546608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.546699 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.546864 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:32:38.547267 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.548016 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.549613 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.550056 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.550206 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.550769 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:32:38.551304 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:32:38.551487 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.551546 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.551635 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 01:32:38.561807 10140 raft_consensus.cc:1261] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 1 FOLLOWER]: Refusing update from remote peer faed72eb6aff47da88f59700c3c3666b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:32:38.562012 10006 raft_consensus.cc:1261] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 1 FOLLOWER]: Refusing update from remote peer faed72eb6aff47da88f59700c3c3666b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:32:38.563067 10212 consensus_queue.cc:1035] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [LEADER]: Connected to new peer: Peer: permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:32:38.563411 10215 consensus_queue.cc:1035] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [LEADER]: Connected to new peer: Peer: permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:32:38.577306 10220 mvcc.cc:205] Tried to move back new op lower bound from 7333068630255984640 to 7333068629952696320. Current Snapshot: MvccSnapshot[applied={T|T < 7333068630255984640}]
I20260925 01:32:38.590402 9727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:32:38.590643 9727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:32:38.594515 9727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Sending DeleteTablet for 3 replicas of tablet 82f1c6959c094577b6cb3bdc6b2b24a1
I20260925 01:32:38.595831 10120 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f1c6959c094577b6cb3bdc6b2b24a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:38 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:53622
I20260925 01:32:38.595850 9986 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f1c6959c094577b6cb3bdc6b2b24a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:38 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:48042
I20260925 01:32:38.595913 9852 tablet_service.cc:1514] Processing DeleteTablet for tablet 82f1c6959c094577b6cb3bdc6b2b24a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:38 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:53730
I20260925 01:32:38.600782 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 9789
I20260925 01:32:38.606194 10229 tablet_replica.cc:330] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: stopping tablet replica
I20260925 01:32:38.606746 10229 raft_consensus.cc:2229] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:32:38.607511 10229 raft_consensus.cc:2258] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:38.610725 10227 tablet_replica.cc:330] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: stopping tablet replica
I20260925 01:32:38.611268 10227 raft_consensus.cc:2229] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:32:38.611826 10227 raft_consensus.cc:2258] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:38.613186 10229 ts_tablet_manager.cc:1905] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:38.623735 10227 ts_tablet_manager.cc:1905] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:38.624497 10228 tablet_replica.cc:330] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: stopping tablet replica
Sep 25 01:32:38 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:32:38.624980 10229 ts_tablet_manager.cc:1918] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:38.625108 10228 raft_consensus.cc:2229] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:32:38.625319 10229 log.cc:1196] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/82f1c6959c094577b6cb3bdc6b2b24a1
I20260925 01:32:38.625756 10228 raft_consensus.cc:2258] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:38.626109 10229 ts_tablet_manager.cc:1939] T 82f1c6959c094577b6cb3bdc6b2b24a1 P faed72eb6aff47da88f59700c3c3666b: Deleting consensus metadata
I20260925 01:32:38.628279 9712 catalog_manager.cc:4841] TS faed72eb6aff47da88f59700c3c3666b (127.27.160.1:33579): tablet 82f1c6959c094577b6cb3bdc6b2b24a1 (table test-table [id=9b191d6030d84c6d976d1653f8b25da4]) successfully deleted
I20260925 01:32:38.630218 10228 ts_tablet_manager.cc:1905] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:38.634531 10227 ts_tablet_manager.cc:1918] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:38.634735 10227 log.cc:1196] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/82f1c6959c094577b6cb3bdc6b2b24a1
I20260925 01:32:38.635358 10227 ts_tablet_manager.cc:1939] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 054e01f6aa2347c890e8f4e834006bef: Deleting consensus metadata
I20260925 01:32:38.635720 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.602685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.27.160.1:33579)
0925 01:32:38.602940 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.602949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.608601 (+ 5652us) server_negotiation.cc:407] Connection header received
0925 01:32:38.608863 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.608871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.609025 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.609213 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:38.612660 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.614499 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.617669 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.618776 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.618953 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.627031 (+ 8078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:38.627044 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:38.627062 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:38.627116 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:38.631490 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.632649 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.632665 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.632668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.632892 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.633508 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.633514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.633516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.634000 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:38.634323 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.635008 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.635289 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:32:38.637403 9912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:38.637542 9713 catalog_manager.cc:4841] TS 054e01f6aa2347c890e8f4e834006bef (127.27.160.3:37535): tablet 82f1c6959c094577b6cb3bdc6b2b24a1 (table test-table [id=9b191d6030d84c6d976d1653f8b25da4]) successfully deleted
I20260925 01:32:38.638618 10228 ts_tablet_manager.cc:1918] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:38.638809 10228 log.cc:1196] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/82f1c6959c094577b6cb3bdc6b2b24a1
I20260925 01:32:38.639355 10228 ts_tablet_manager.cc:1939] T 82f1c6959c094577b6cb3bdc6b2b24a1 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Deleting consensus metadata
I20260925 01:32:38.641376 9715 catalog_manager.cc:4841] TS 582e38dd0b584d5a8f4f8bf79ad00c1e (127.27.160.2:46021): tablet 82f1c6959c094577b6cb3bdc6b2b24a1 (table test-table [id=9b191d6030d84c6d976d1653f8b25da4]) successfully deleted
I20260925 01:32:38.740093 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 9789
I20260925 01:32:38.759603 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 9923
Sep 25 01:32:38 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:32:38.778362 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.760471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.27.160.2:46021)
0925 01:32:38.760678 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.760769 (+ 91us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.760796 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:32:38.760990 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.760995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.761080 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.761372 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:38.762874 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.763731 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.765489 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.765952 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.766108 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.771338 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:38.771356 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:38.771374 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:38.771419 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:38.774921 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.775842 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.775851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.775854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.776004 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.776468 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.776475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.776478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.776951 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:38.777316 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.777686 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.777949 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:32:38.779134 10046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:38.879312 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 9923
I20260925 01:32:38.896714 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 10057
Sep 25 01:32:38 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:32:38.914968 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:38.897664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.27.160.3:37535)
0925 01:32:38.897880 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:38.897887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:38.897977 (+ 90us) server_negotiation.cc:407] Connection header received
0925 01:32:38.898201 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:38.898207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:38.898302 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:38.898446 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:38.900017 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.900751 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.902352 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:38.902758 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:38.902904 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:38.907998 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:38.908015 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:38.908032 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:38.908077 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:38.911557 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.912404 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.912423 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.912426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.912603 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:38.913100 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:38.913108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:38.913161 (+ 53us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:38.913673 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:38.914038 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:38.914346 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 01:32:38.914595 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 01:32:38.915640 10180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:39.007320 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 10057
I20260925 01:32:39.025349 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 9696
I20260925 01:32:39.026594 9758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:39.140759 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 9696
I20260925 01:32:39.163633 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40651
--webserver_interface=127.27.160.62
--webserver_port=39581
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:40651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:39.518180 10235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:39.518520 10235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:39.518702 10235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:39.526408 10235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:39.526553 10235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:39.526647 10235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:39.526753 10235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:39.526819 10235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:39.539461 10235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:40651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=39581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:39.541599 10235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:39.543231 10235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:39.571903 10241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:39.571885 10242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:39.571825 10244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:39.575225 10235 server_base.cc:1034] running on GCE node
I20260925 01:32:39.576283 10235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:39.578217 10235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:39.579562 10235 hybrid_clock.cc:648] HybridClock initialized: now 1790299959579485 us; error 67 us; skew 500 ppm
I20260925 01:32:39.580123 10235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:39 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299959, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:39.588810 10235 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:32:39.591035 10235 webserver.cc:460] Webserver started at http://127.27.160.62:39581/ using document root <none> and password file <none>
I20260925 01:32:39.591914 10235 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:39.592098 10235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:39.598789 10235 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:32:39.602392 10251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:39.604074 10235 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:32:39.604472 10235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "2387cfabf2d84795915ecb496461ee14"
format_stamp: "Formatted at 2026-09-25 01:32:36 on dist-test-slave-h6hf"
server_key: "fddf8d78b5a8f5cca68544f02215195e"
server_key_iv: "603885e2c5b98993f4852961065c7f66"
server_key_version: "encryptionkey@0"
I20260925 01:32:39.605662 10235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:39.628808 10235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:39.634505 10235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:39.634788 10235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:39.656394 10235 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:40651
I20260925 01:32:39.656426 10302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:40651 every 8 connection(s)
I20260925 01:32:39.658257 10235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:32:39.661407 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 10235
I20260925 01:32:39.662470 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:33579
--local_ip_for_outbound_sockets=127.27.160.1
--tserver_master_addrs=127.27.160.62:40651
--webserver_port=44945
--webserver_interface=127.27.160.1
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:32:39.669060 10303 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:32:39.674256 10303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Bootstrap starting.
I20260925 01:32:39.695652 10303 log.cc:824] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:39.711331 10303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:32:39.712129 10303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Bootstrap complete.
I20260925 01:32:39.723392 10303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:39.724418 10303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2387cfabf2d84795915ecb496461ee14, State: Initialized, Role: FOLLOWER
I20260925 01:32:39.725353 10303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:39.725688 10303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:32:39.725844 10303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:32:39.726078 10303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:32:39.730885 10303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:39.731717 10303 leader_election.cc:302] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2387cfabf2d84795915ecb496461ee14; no voters:
I20260925 01:32:39.732578 10303 leader_election.cc:288] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:32:39.732758 10307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:32:39.734627 10307 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 2 LEADER]: Becoming Leader. State: Replica: 2387cfabf2d84795915ecb496461ee14, State: Running, Role: LEADER
I20260925 01:32:39.735311 10307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:39.735913 10303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:32:39.738412 10309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2387cfabf2d84795915ecb496461ee14. Latest consensus state: current_term: 2 leader_uuid: "2387cfabf2d84795915ecb496461ee14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } } }
I20260925 01:32:39.738494 10308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2387cfabf2d84795915ecb496461ee14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } } }
I20260925 01:32:39.738881 10309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:39.738977 10308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:39.758293 10313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:32:39.769941 10313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:32:39.772279 10313 catalog_manager.cc:1259] Loaded cluster ID: 286a907ab3954d24bf765f3be254655e
I20260925 01:32:39.772459 10313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:32:39.777045 10313 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 01:32:39.796562 10313 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 01:32:39.804494 10313 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:32:39.806138 10313 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Loaded TSK: 0
I20260925 01:32:39.807390 10313 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:32:40.073019 10305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:40.073451 10305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:40.073665 10305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:40.081964 10305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:40.082089 10305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:40.082266 10305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:32:40.092584 10305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:33579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=44945
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:40.094204 10305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:40.095582 10305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:40.109875 10331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:40.109989 10330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:40.110498 10333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:40.111423 10305 server_base.cc:1034] running on GCE node
I20260925 01:32:40.112133 10305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:40.113269 10305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:40.114517 10305 hybrid_clock.cc:648] HybridClock initialized: now 1790299960114452 us; error 63 us; skew 500 ppm
I20260925 01:32:40.114864 10305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:40 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:40.118530 10305 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:32:40.120208 10305 webserver.cc:460] Webserver started at http://127.27.160.1:44945/ using document root <none> and password file <none>
I20260925 01:32:40.120970 10305 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:40.121194 10305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:40.126933 10305 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260925 01:32:40.130088 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:40.131718 10305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:32:40.131909 10305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "faed72eb6aff47da88f59700c3c3666b"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1f0605ff5d6262d29c6ee0250c74207e"
server_key_iv: "33c25efc564ce326605587391e8cbbf8"
server_key_version: "encryptionkey@0"
I20260925 01:32:40.132956 10305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:40.158361 10305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:40.163375 10305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:40.163618 10305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:40.164609 10305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:40.166199 10305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:40.166294 10305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:40.166369 10305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:40.166435 10305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:40.201327 10305 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:33579
I20260925 01:32:40.201481 10452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:33579 every 8 connection(s)
I20260925 01:32:40.202872 10305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:32:40.212321 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 10305
I20260925 01:32:40.213004 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:46021
--local_ip_for_outbound_sockets=127.27.160.2
--tserver_master_addrs=127.27.160.62:40651
--webserver_port=39567
--webserver_interface=127.27.160.2
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:32:40 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:40.228510 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:40.206180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:49713)
0925 01:32:40.206915 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:40.206952 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:40.208710 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:40.210580 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:40.210592 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:40.211236 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:40.211828 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:40.211849 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:40.213793 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:40.213798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:40.215063 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:40.215078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:40.216201 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:40.216209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:40.216517 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:40.217278 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:40.217305 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:40.219641 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:40.223566 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:40.223576 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:40.223597 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:40.224103 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:40.224644 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:40.224648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:40.224653 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:40.224873 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:40.225769 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:40.225779 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:40.226185 (+ 406us) client_negotiation.cc:769] Sending connection context
0925 01:32:40.226481 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 01:32:40.227104 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":276,"threads_started":1}
I20260925 01:32:40.228510 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:40.205829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:49713 (local address 127.27.160.62:40651)
0925 01:32:40.206394 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:40.206407 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:40.207086 (+ 679us) server_negotiation.cc:407] Connection header received
0925 01:32:40.209493 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:40.209511 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:40.209988 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:40.210402 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:40.212081 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:40.213586 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:40.215228 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:40.215943 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:40.216166 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:40.219979 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:40.219988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:40.220006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:40.220048 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:40.223372 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:40.224281 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:40.224296 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:40.224303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:40.224498 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:40.225002 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:40.225008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:40.225013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:40.225606 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:40.225873 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:40.226717 (+ 844us) server_negotiation.cc:299] Negotiation successful
0925 01:32:40.227106 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260925 01:32:40.231635 10453 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40651
I20260925 01:32:40.232053 10453 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:40.233108 10453 heartbeater.cc:511] Master 127.27.160.62:40651 requested a full tablet report, sending...
I20260925 01:32:40.235757 10268 ts_manager.cc:194] Registered new tserver with Master: faed72eb6aff47da88f59700c3c3666b (127.27.160.1:33579)
I20260925 01:32:40.237757 10268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:49713
W20260925 01:32:40.563994 10457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:40.564316 10457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:40.564414 10457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:40.572084 10457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:40.572221 10457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:40.572337 10457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:32:40.582659 10457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:46021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=39567
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:40.583983 10457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:40.585378 10457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:40.599855 10463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:40.599887 10465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:40.600003 10462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:40.601541 10457 server_base.cc:1034] running on GCE node
I20260925 01:32:40.602298 10457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:40.603767 10457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:40.605002 10457 hybrid_clock.cc:648] HybridClock initialized: now 1790299960604902 us; error 111 us; skew 500 ppm
I20260925 01:32:40.605528 10457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:40 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:40.609848 10457 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:32:40.612246 10457 webserver.cc:460] Webserver started at http://127.27.160.2:39567/ using document root <none> and password file <none>
I20260925 01:32:40.613026 10457 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:40.613212 10457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:40.619478 10457 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 01:32:40.622678 10472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:40.624361 10457 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:32:40.624522 10457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1feece858f52b0609bdd2b37cd9a2ce3"
server_key_iv: "736a2052663b539dbe3eb6cc97e145f2"
server_key_version: "encryptionkey@0"
I20260925 01:32:40.625650 10457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:40.655704 10457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:40.661829 10457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:40.662128 10457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:40.663347 10457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:40.665078 10457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:40.665215 10457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 01:32:40.665318 10457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:40.665393 10457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:40.700845 10457 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:46021
I20260925 01:32:40.701067 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:46021 every 8 connection(s)
I20260925 01:32:40.702381 10457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:32:40.709483 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 10457
I20260925 01:32:40.710204 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:37535
--local_ip_for_outbound_sockets=127.27.160.3
--tserver_master_addrs=127.27.160.62:40651
--webserver_port=42139
--webserver_interface=127.27.160.3
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:32:40 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299960, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:40.724279 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:40.705308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:39465 (local address 127.27.160.62:40651)
0925 01:32:40.705554 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:40.705561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:40.706955 (+ 1394us) server_negotiation.cc:407] Connection header received
0925 01:32:40.708831 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:40.708837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:40.709019 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:40.709154 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:40.711066 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:40.711782 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:40.713538 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:40.713992 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:40.714140 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:40.718184 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:40.718189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:40.718192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:40.718229 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:40.720869 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:40.721773 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:40.721778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:40.721780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:40.721909 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:40.722382 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:40.722386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:40.722388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:40.722701 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:40.722875 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:40.723650 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 01:32:40.723934 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:32:40.725440 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:40.705928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:39465)
0925 01:32:40.706734 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:40.706790 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:40.708524 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:40.709447 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:40.709456 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:40.710154 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:40.710892 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:40.710913 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:40.711951 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:40.711963 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:40.713347 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:40.713360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:40.714241 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:40.714248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:40.714503 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:40.715282 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:40.715306 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:40.717886 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:40.721039 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:40.721049 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:40.721068 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:40.721614 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:40.722028 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:40.722033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:40.722036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:40.722247 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:40.722852 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:40.722860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:40.723253 (+ 393us) client_negotiation.cc:769] Sending connection context
0925 01:32:40.723538 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 01:32:40.724128 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":308,"threads_started":1}
I20260925 01:32:40.727339 10585 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40651
I20260925 01:32:40.727746 10585 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:40.728798 10585 heartbeater.cc:511] Master 127.27.160.62:40651 requested a full tablet report, sending...
I20260925 01:32:40.730868 10268 ts_manager.cc:194] Registered new tserver with Master: 582e38dd0b584d5a8f4f8bf79ad00c1e (127.27.160.2:46021)
I20260925 01:32:40.732237 10268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:39465
W20260925 01:32:41.059392 10588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:41.059895 10588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:41.060014 10588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:41.067607 10588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:41.067744 10588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:41.067862 10588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:32:41.078190 10588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:37535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=42139
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:41.079563 10588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:41.080909 10588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:41.095155 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:41.095391 10593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:41.095325 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:41.096239 10588 server_base.cc:1034] running on GCE node
I20260925 01:32:41.096940 10588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:41.098034 10588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:41.099252 10588 hybrid_clock.cc:648] HybridClock initialized: now 1790299961099195 us; error 45 us; skew 500 ppm
I20260925 01:32:41.099572 10588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:41 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299961, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:41.103001 10588 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:32:41.104542 10588 webserver.cc:460] Webserver started at http://127.27.160.3:42139/ using document root <none> and password file <none>
I20260925 01:32:41.105456 10588 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:41.105629 10588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:41.111696 10588 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:32:41.114954 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:41.116583 10588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:32:41.116783 10588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "054e01f6aa2347c890e8f4e834006bef"
format_stamp: "Formatted at 2026-09-25 01:32:38 on dist-test-slave-h6hf"
server_key: "1709202b120096b14f5079b7eaeaeb80"
server_key_iv: "fe84fcffaf22cae1b5bf94f335e7a7b9"
server_key_version: "encryptionkey@0"
I20260925 01:32:41.117944 10588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:41.136428 10588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:41.141546 10588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:41.141847 10588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:41.142951 10588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:41.144484 10588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:41.144595 10588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:41.144739 10588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:41.144826 10588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:41.180156 10588 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:37535
I20260925 01:32:41.180284 10715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:37535 every 8 connection(s)
I20260925 01:32:41.181674 10588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:32:41.187361 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 10588
Sep 25 01:32:41 dist-test-slave-h6hf krb5kdc[9683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299961, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:32:41.204555 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.184451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:60667 (local address 127.27.160.62:40651)
0925 01:32:41.184671 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.184684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.185894 (+ 1210us) server_negotiation.cc:407] Connection header received
0925 01:32:41.188233 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.188240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.188370 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.188489 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:41.190679 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.191410 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.193299 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.193804 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.193944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.198032 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:41.198043 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:41.198046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:41.198104 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:41.201238 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:41.202045 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:41.202051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:41.202055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:41.202189 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:41.202676 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:41.202681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:41.202684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:41.202998 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:41.203162 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.203945 (+ 783us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.204197 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"spinlock_wait_cycles":3456}
I20260925 01:32:41.205935 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.184928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:60667)
0925 01:32:41.185722 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:41.185767 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:41.187879 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:41.188844 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:41.188862 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:41.189872 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:32:41.190495 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.190513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.191570 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.191575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.193107 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.193144 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.194047 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.194054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.194407 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.195249 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:32:41.195284 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:32:41.197778 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:32:41.201402 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:41.201412 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:41.201430 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:41.201901 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:41.202312 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:32:41.202317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:32:41.202321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:32:41.202539 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:32:41.203135 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:32:41.203143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:32:41.203530 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 01:32:41.203811 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 01:32:41.204403 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":370,"threads_started":1}
I20260925 01:32:41.207693 10716 heartbeater.cc:347] Connected to a master server at 127.27.160.62:40651
I20260925 01:32:41.208170 10716 heartbeater.cc:464] Registering TS with master...
I20260925 01:32:41.208889 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.188469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.27.160.62:40651)
0925 01:32:41.189994 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.189999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.190013 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:32:41.190056 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.190060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.190112 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.190230 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:41.191741 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.192512 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.194262 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.194898 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.194988 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.198026 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:41.198032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:41.198035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:41.198067 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:41.201632 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:41.206364 (+ 4732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:41.206369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:41.206371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:41.206550 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:41.207174 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:41.207183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:41.207186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:41.207534 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:41.207843 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.208207 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.208472 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1391,"thread_start_us":1207,"threads_started":1}
I20260925 01:32:41.209175 10716 heartbeater.cc:511] Master 127.27.160.62:40651 requested a full tablet report, sending...
I20260925 01:32:41.210949 10267 ts_manager.cc:194] Registered new tserver with Master: 054e01f6aa2347c890e8f4e834006bef (127.27.160.3:37535)
I20260925 01:32:41.211898 10267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:60667
I20260925 01:32:41.224649 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:32:41.232712 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.225758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.27.160.62:40651)
0925 01:32:41.225991 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.225996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.226506 (+ 510us) server_negotiation.cc:407] Connection header received
0925 01:32:41.226773 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.226777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.226927 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.227073 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:32:41.227561 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.228309 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.230419 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.230869 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.230992 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.231596 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:32:41.232130 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:32:41.232242 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.232426 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.232504 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 01:32:41.233952 10267 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 01:32:41.241367 10453 heartbeater.cc:503] Master 127.27.160.62:40651 was elected leader, sending a full tablet report...
W20260925 01:32:41.242839 10267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:32:41.280776 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.257226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.27.160.3:37535)
0925 01:32:41.271110 (+ 13884us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.271122 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.271166 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:32:41.271298 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.271309 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.271642 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.272164 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:41.274669 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.276176 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.278281 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.279384 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.279563 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.279853 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.280215 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.280399 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13695,"thread_start_us":13518,"threads_started":1}
I20260925 01:32:41.280997 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.256966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:37535 (local address 127.0.0.1:39008)
0925 01:32:41.268776 (+ 11810us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:41.268804 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:41.268983 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:41.271959 (+ 2976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:41.271968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:41.272047 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:41.273362 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.273380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.276349 (+ 2969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.276357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.278094 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.278108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.279637 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.279644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.279928 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.279980 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 01:32:41.280133 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 01:32:41.280368 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11605,"mutex_wait_us":99,"spinlock_wait_cycles":38528,"thread_start_us":12097,"threads_started":1}
I20260925 01:32:41.282084 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.257482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.27.160.1:33579)
0925 01:32:41.270594 (+ 13112us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.270606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.270644 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:32:41.270781 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.270793 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.271201 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.271485 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:41.273665 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.275406 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.278432 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.280205 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.280359 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.280604 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.281570 (+ 966us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.281754 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12924,"thread_start_us":12738,"threads_started":1}
I20260925 01:32:41.282137 10731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.256965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:33579 (local address 127.0.0.1:42392)
0925 01:32:41.268013 (+ 11048us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:41.268064 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:41.268299 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:41.271942 (+ 3643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:41.271956 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:41.272037 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:41.272656 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.272677 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.275609 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.275616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.278270 (+ 2654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.278285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.280803 (+ 2518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.280811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.281354 (+ 543us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.281380 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:32:41.281512 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 01:32:41.281677 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10738,"mutex_wait_us":351,"spinlock_wait_cycles":6912,"thread_start_us":17403,"threads_started":1}
I20260925 01:32:41.285010 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.260902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.27.160.2:46021)
0925 01:32:41.275011 (+ 14109us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.275021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.275052 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:32:41.275150 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.275158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.275478 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.275742 (+ 264us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:41.276406 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.279404 (+ 2998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.282371 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.283882 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.284043 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.284311 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.284554 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.284723 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13934,"thread_start_us":13818,"threads_started":1}
I20260925 01:32:41.285020 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.260838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:46021 (local address 127.0.0.1:51532)
0925 01:32:41.274152 (+ 13314us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:41.274173 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:41.274292 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:41.275727 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:41.275736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:41.275768 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:41.276205 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.276214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.279615 (+ 3401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.279621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.282129 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.282146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.284144 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.284151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.284342 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.284378 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:32:41.284484 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 01:32:41.284622 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13150,"mutex_wait_us":4,"thread_start_us":6659,"threads_started":1}
I20260925 01:32:41.286309 10651 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9c587803764a44aa46d445b81871b0 (DEFAULT_TABLE table=test-table [id=62ca81d5510140c0a29a603bde97492b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:41.286281 10388 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9c587803764a44aa46d445b81871b0 (DEFAULT_TABLE table=test-table [id=62ca81d5510140c0a29a603bde97492b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:41.287645 10651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9c587803764a44aa46d445b81871b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:41.289175 10388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9c587803764a44aa46d445b81871b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:41.289345 10520 tablet_service.cc:1467] Processing CreateTablet for tablet 2e9c587803764a44aa46d445b81871b0 (DEFAULT_TABLE table=test-table [id=62ca81d5510140c0a29a603bde97492b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:32:41.290781 10520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e9c587803764a44aa46d445b81871b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:32:41.306437 10732 tablet_bootstrap.cc:492] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Bootstrap starting.
I20260925 01:32:41.309038 10733 tablet_bootstrap.cc:492] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Bootstrap starting.
I20260925 01:32:41.310583 10732 tablet_bootstrap.cc:654] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:41.312219 10732 log.cc:824] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:41.312494 10734 tablet_bootstrap.cc:492] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Bootstrap starting.
I20260925 01:32:41.313478 10733 tablet_bootstrap.cc:654] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:41.315344 10733 log.cc:824] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:41.316124 10734 tablet_bootstrap.cc:654] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Neither blocks nor log segments found. Creating new log.
I20260925 01:32:41.317691 10734 log.cc:824] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:41.319321 10733 tablet_bootstrap.cc:492] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: No bootstrap required, opened a new log
I20260925 01:32:41.319340 10732 tablet_bootstrap.cc:492] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: No bootstrap required, opened a new log
I20260925 01:32:41.319628 10733 ts_tablet_manager.cc:1397] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 01:32:41.319644 10732 ts_tablet_manager.cc:1397] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260925 01:32:41.321223 10734 tablet_bootstrap.cc:492] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: No bootstrap required, opened a new log
I20260925 01:32:41.321545 10734 ts_tablet_manager.cc:1397] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 01:32:41.327862 10733 raft_consensus.cc:348] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.328341 10733 raft_consensus.cc:374] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:41.328444 10733 raft_consensus.cc:729] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582e38dd0b584d5a8f4f8bf79ad00c1e, State: Initialized, Role: FOLLOWER
I20260925 01:32:41.329295 10733 consensus_queue.cc:260] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.330870 10585 heartbeater.cc:503] Master 127.27.160.62:40651 was elected leader, sending a full tablet report...
I20260925 01:32:41.331477 10733 ts_tablet_manager.cc:1428] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260925 01:32:41.332166 10732 raft_consensus.cc:348] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.332655 10732 raft_consensus.cc:374] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:41.332791 10732 raft_consensus.cc:729] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054e01f6aa2347c890e8f4e834006bef, State: Initialized, Role: FOLLOWER
I20260925 01:32:41.332947 10734 raft_consensus.cc:348] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.333482 10734 raft_consensus.cc:374] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:32:41.333591 10734 raft_consensus.cc:729] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faed72eb6aff47da88f59700c3c3666b, State: Initialized, Role: FOLLOWER
I20260925 01:32:41.333514 10732 consensus_queue.cc:260] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.334182 10734 consensus_queue.cc:260] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.334888 10716 heartbeater.cc:503] Master 127.27.160.62:40651 was elected leader, sending a full tablet report...
I20260925 01:32:41.335551 10732 ts_tablet_manager.cc:1428] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260925 01:32:41.338317 10734 ts_tablet_manager.cc:1428] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260925 01:32:41.433470 10717 tablet.cc:2369] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:32:41.454507 10586 tablet.cc:2369] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:32:41.455381 10454 tablet.cc:2369] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:32:41.628319 10739 raft_consensus.cc:482] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:32:41.628837 10739 raft_consensus.cc:504] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.631171 10739 leader_election.cc:288] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faed72eb6aff47da88f59700c3c3666b (127.27.160.1:33579), 582e38dd0b584d5a8f4f8bf79ad00c1e (127.27.160.2:46021)
I20260925 01:32:41.638590 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.631555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:46021 (local address 127.27.160.3:59505)
0925 01:32:41.631916 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:41.631941 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:41.632200 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:41.632624 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:41.632628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:41.632772 (+ 144us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:41.633014 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.633029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.634306 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.634313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.636048 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.636063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.637592 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.637599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.637845 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.637884 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:32:41.637997 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:32:41.638180 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":26}
I20260925 01:32:41.638653 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.631674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:59505 (local address 127.27.160.2:46021)
0925 01:32:41.631905 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.631911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.632064 (+ 153us) server_negotiation.cc:407] Connection header received
0925 01:32:41.632380 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.632385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.632480 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.632580 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:41.633213 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.634165 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.636220 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.637430 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.637573 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.637633 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.638079 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.638296 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:32:41.639426 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.631725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:46703 (local address 127.27.160.1:33579)
0925 01:32:41.631931 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.631939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.632709 (+ 770us) server_negotiation.cc:407] Connection header received
0925 01:32:41.632805 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.632811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.632897 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.633016 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:32:41.633780 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.634711 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.636815 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.638135 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.638300 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.638391 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.638909 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.639068 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:32:41.640283 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.631629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:33579 (local address 127.27.160.3:46703)
0925 01:32:41.632390 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:41.632409 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:32:41.632566 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:32:41.633038 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:32:41.633041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:32:41.633062 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:32:41.633580 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.633597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.634892 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.634897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.636578 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:32:41.636595 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.638431 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.638439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:32:41.638660 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.638705 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 01:32:41.639462 (+ 757us) client_negotiation.cc:240] Negotiation successful
0925 01:32:41.639825 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":3,"thread_start_us":380,"threads_started":1}
I20260925 01:32:41.641179 10540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9c587803764a44aa46d445b81871b0" candidate_uuid: "054e01f6aa2347c890e8f4e834006bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" is_pre_election: true
I20260925 01:32:41.641556 10408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9c587803764a44aa46d445b81871b0" candidate_uuid: "054e01f6aa2347c890e8f4e834006bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faed72eb6aff47da88f59700c3c3666b" is_pre_election: true
I20260925 01:32:41.641780 10540 raft_consensus.cc:2454] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 054e01f6aa2347c890e8f4e834006bef in term 0.
I20260925 01:32:41.642102 10408 raft_consensus.cc:2454] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 054e01f6aa2347c890e8f4e834006bef in term 0.
I20260925 01:32:41.642722 10607 leader_election.cc:302] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054e01f6aa2347c890e8f4e834006bef, 582e38dd0b584d5a8f4f8bf79ad00c1e; no voters:
I20260925 01:32:41.643239 10739 raft_consensus.cc:2781] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:32:41.643416 10739 raft_consensus.cc:482] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:32:41.643535 10739 raft_consensus.cc:3037] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:41.646119 10739 raft_consensus.cc:504] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.646859 10739 leader_election.cc:288] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [CANDIDATE]: Term 1 election: Requested vote from peers faed72eb6aff47da88f59700c3c3666b (127.27.160.1:33579), 582e38dd0b584d5a8f4f8bf79ad00c1e (127.27.160.2:46021)
I20260925 01:32:41.647549 10540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9c587803764a44aa46d445b81871b0" candidate_uuid: "054e01f6aa2347c890e8f4e834006bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e"
I20260925 01:32:41.647629 10408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e9c587803764a44aa46d445b81871b0" candidate_uuid: "054e01f6aa2347c890e8f4e834006bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faed72eb6aff47da88f59700c3c3666b"
I20260925 01:32:41.647766 10540 raft_consensus.cc:3037] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:41.647925 10408 raft_consensus.cc:3037] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:32:41.650590 10540 raft_consensus.cc:2454] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 054e01f6aa2347c890e8f4e834006bef in term 1.
I20260925 01:32:41.650624 10408 raft_consensus.cc:2454] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 054e01f6aa2347c890e8f4e834006bef in term 1.
I20260925 01:32:41.651227 10607 leader_election.cc:302] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 054e01f6aa2347c890e8f4e834006bef, 582e38dd0b584d5a8f4f8bf79ad00c1e; no voters:
I20260925 01:32:41.651700 10739 raft_consensus.cc:2781] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:32:41.652225 10739 raft_consensus.cc:686] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 1 LEADER]: Becoming Leader. State: Replica: 054e01f6aa2347c890e8f4e834006bef, State: Running, Role: LEADER
I20260925 01:32:41.652892 10739 consensus_queue.cc:237] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } }
I20260925 01:32:41.658557 10267 catalog_manager.cc:5495] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 054e01f6aa2347c890e8f4e834006bef (127.27.160.3). New cstate: current_term: 1 leader_uuid: "054e01f6aa2347c890e8f4e834006bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "054e01f6aa2347c890e8f4e834006bef" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 37535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 } health_report { overall_health: UNKNOWN } } }
I20260925 01:32:41.745474 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.739101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.27.160.3:37535)
0925 01:32:41.739310 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.739314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.739809 (+ 495us) server_negotiation.cc:407] Connection header received
0925 01:32:41.739990 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.739994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.740085 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.740229 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:32:41.740709 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.741602 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.743233 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.743621 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.743714 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.744330 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:32:41.744852 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:32:41.744982 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.745087 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.745214 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:32:41.754312 10540 raft_consensus.cc:1261] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 1 FOLLOWER]: Refusing update from remote peer 054e01f6aa2347c890e8f4e834006bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:32:41.754340 10408 raft_consensus.cc:1261] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 1 FOLLOWER]: Refusing update from remote peer 054e01f6aa2347c890e8f4e834006bef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:32:41.755362 10742 consensus_queue.cc:1035] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "faed72eb6aff47da88f59700c3c3666b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 33579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:32:41.755725 10739 consensus_queue.cc:1035] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:32:41.775723 10747 mvcc.cc:205] Tried to move back new op lower bound from 7333068643334909952 to 7333068642935640064. Current Snapshot: MvccSnapshot[applied={T|T < 7333068643334909952 or (T in {7333068643334909952})}]
I20260925 01:32:41.784390 10268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:32:41.784624 10268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:32:41.788408 10268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Sending DeleteTablet for 3 replicas of tablet 2e9c587803764a44aa46d445b81871b0
I20260925 01:32:41.789769 10388 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9c587803764a44aa46d445b81871b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:41 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:42392
I20260925 01:32:41.789933 10520 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9c587803764a44aa46d445b81871b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:41 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:51532
I20260925 01:32:41.789947 10651 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e9c587803764a44aa46d445b81871b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:32:41 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:39008
I20260925 01:32:41.791296 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 10305
I20260925 01:32:41.802242 10756 tablet_replica.cc:330] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: stopping tablet replica
I20260925 01:32:41.802963 10756 raft_consensus.cc:2229] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:32:41.803682 10756 raft_consensus.cc:2258] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:41.805565 10758 tablet_replica.cc:330] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: stopping tablet replica
I20260925 01:32:41.805802 10757 tablet_replica.cc:330] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: stopping tablet replica
I20260925 01:32:41.806030 10758 raft_consensus.cc:2229] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:32:41.806212 10757 raft_consensus.cc:2229] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:32:41.806694 10757 raft_consensus.cc:2258] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:41.806751 10758 raft_consensus.cc:2258] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:32:41.810153 10756 ts_tablet_manager.cc:1905] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:41.809077 10757 ts_tablet_manager.cc:1905] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:41.813006 10758 ts_tablet_manager.cc:1905] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:32:41.817502 10756 ts_tablet_manager.cc:1918] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:41.817687 10756 log.cc:1196] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/2e9c587803764a44aa46d445b81871b0
I20260925 01:32:41.818251 10756 ts_tablet_manager.cc:1939] T 2e9c587803764a44aa46d445b81871b0 P faed72eb6aff47da88f59700c3c3666b: Deleting consensus metadata
I20260925 01:32:41.819197 10757 ts_tablet_manager.cc:1918] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:41.819453 10757 log.cc:1196] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/2e9c587803764a44aa46d445b81871b0
I20260925 01:32:41.820300 10757 ts_tablet_manager.cc:1939] T 2e9c587803764a44aa46d445b81871b0 P 582e38dd0b584d5a8f4f8bf79ad00c1e: Deleting consensus metadata
I20260925 01:32:41.820307 10252 catalog_manager.cc:4841] TS faed72eb6aff47da88f59700c3c3666b (127.27.160.1:33579): tablet 2e9c587803764a44aa46d445b81871b0 (table test-table [id=62ca81d5510140c0a29a603bde97492b]) successfully deleted
I20260925 01:32:41.823819 10255 catalog_manager.cc:4841] TS 582e38dd0b584d5a8f4f8bf79ad00c1e (127.27.160.2:46021): tablet 2e9c587803764a44aa46d445b81871b0 (table test-table [id=62ca81d5510140c0a29a603bde97492b]) successfully deleted
I20260925 01:32:41.824700 10758 ts_tablet_manager.cc:1918] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:32:41.824903 10758 log.cc:1196] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/2e9c587803764a44aa46d445b81871b0
I20260925 01:32:41.825459 10758 ts_tablet_manager.cc:1939] T 2e9c587803764a44aa46d445b81871b0 P 054e01f6aa2347c890e8f4e834006bef: Deleting consensus metadata
I20260925 01:32:41.827241 10253 catalog_manager.cc:4841] TS 054e01f6aa2347c890e8f4e834006bef (127.27.160.3:37535): tablet 2e9c587803764a44aa46d445b81871b0 (table test-table [id=62ca81d5510140c0a29a603bde97492b]) successfully deleted
I20260925 01:32:41.828438 10729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.792192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.27.160.1:33579)
0925 01:32:41.792401 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.792410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.804770 (+ 12360us) server_negotiation.cc:407] Connection header received
0925 01:32:41.805026 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.805033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.805177 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.805320 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:41.809269 (+ 3949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.810393 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.812713 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.813423 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.813551 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.820943 (+ 7392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:41.820959 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:41.820978 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:41.821028 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:41.824950 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:41.825931 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:41.825942 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:41.825947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:41.826119 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:41.826613 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:41.826619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:41.826622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:41.827126 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:41.827511 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.827865 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.828042 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:32:41.829067 10448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:41.951664 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 10305
I20260925 01:32:41.970394 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 10457
I20260925 01:32:41.987933 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:41.971386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.27.160.2:46021)
0925 01:32:41.971605 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:41.971612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:41.971635 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:32:41.971892 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:41.971898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:41.971985 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:41.972095 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:41.973802 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.974947 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.976532 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:41.977253 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:41.977388 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:41.980473 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:41.980496 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:41.980519 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:41.980573 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:41.984420 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:41.985363 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:41.985373 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:41.985377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:41.985566 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:41.986050 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:41.986056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:41.986059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:41.986544 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:41.986855 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:41.987278 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 01:32:41.987522 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:32:41.988747 10580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:42.081934 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 10457
I20260925 01:32:42.100673 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 10588
I20260925 01:32:42.119518 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:42.101788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.27.160.3:37535)
0925 01:32:42.101996 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:42.102000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:42.102024 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:32:42.102207 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:42.102214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:42.102329 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:42.102443 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:42.104411 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:42.105508 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:42.107449 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:42.108004 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:42.108131 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:42.111910 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:42.111925 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:42.111942 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:42.111987 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:42.115885 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:42.116918 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:42.116927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:42.116929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:42.117113 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:42.117645 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:42.117649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:42.117652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:42.118080 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:42.118334 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:42.118820 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 01:32:42.119024 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 01:32:42.120451 10711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:42.248462 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 10588
I20260925 01:32:42.270680 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 10235
I20260925 01:32:42.271972 10298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:42.388306 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 10235
I20260925 01:32:42.410933 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40651
--webserver_interface=127.27.160.62
--webserver_port=39581
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:40651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:32:42.773422 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:42.773792 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:42.773993 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:42.782047 10764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:32:42.782193 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:42.782300 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:42.782418 10764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:32:42.782514 10764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:32:42.793195 10764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:40651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:40651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=39581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:42.794785 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:42.796258 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:42.809942 10769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:42.810111 10770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:42.810146 10772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:42.810977 10764 server_base.cc:1034] running on GCE node
I20260925 01:32:42.811923 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:42.813619 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:42.814877 10764 hybrid_clock.cc:648] HybridClock initialized: now 1790299962814802 us; error 68 us; skew 500 ppm
I20260925 01:32:42.815289 10764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:42 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299962, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:42.818759 10764 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:32:42.820420 10764 webserver.cc:460] Webserver started at http://127.27.160.62:39581/ using document root <none> and password file <none>
I20260925 01:32:42.821237 10764 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:42.821399 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:42.827126 10764 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:32:42.830262 10779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:42.831845 10764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:32:42.832057 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "2387cfabf2d84795915ecb496461ee14"
format_stamp: "Formatted at 2026-09-25 01:32:36 on dist-test-slave-h6hf"
server_key: "fddf8d78b5a8f5cca68544f02215195e"
server_key_iv: "603885e2c5b98993f4852961065c7f66"
server_key_version: "encryptionkey@0"
I20260925 01:32:42.833101 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:42.852921 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:42.858188 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:42.858511 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:42.879056 10764 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:40651
I20260925 01:32:42.879076 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:40651 every 8 connection(s)
I20260925 01:32:42.880684 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:32:42.886977 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 10764
I20260925 01:32:42.888049 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:33579
--local_ip_for_outbound_sockets=127.27.160.1
--tserver_master_addrs=127.27.160.62:40651
--webserver_port=44945
--webserver_interface=127.27.160.1
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:32:42.888818 10831 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:32:42.894476 10831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Bootstrap starting.
I20260925 01:32:42.916785 10831 log.cc:824] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Log is configured to *not* fsync() on all Append() calls
I20260925 01:32:42.933970 10831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:32:42.934599 10831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14: Bootstrap complete.
I20260925 01:32:42.944773 10831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:42.945791 10831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2387cfabf2d84795915ecb496461ee14, State: Initialized, Role: FOLLOWER
I20260925 01:32:42.946560 10831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:42.946823 10831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:32:42.946944 10831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:32:42.947144 10831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 2 FOLLOWER]: Advancing to term 3
I20260925 01:32:42.951315 10831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:42.951864 10831 leader_election.cc:302] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2387cfabf2d84795915ecb496461ee14; no voters:
I20260925 01:32:42.952581 10831 leader_election.cc:288] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 01:32:42.952744 10835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 01:32:42.954006 10835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [term 3 LEADER]: Becoming Leader. State: Replica: 2387cfabf2d84795915ecb496461ee14, State: Running, Role: LEADER
I20260925 01:32:42.954666 10835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } }
I20260925 01:32:42.955032 10831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:32:42.957604 10837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2387cfabf2d84795915ecb496461ee14. Latest consensus state: current_term: 3 leader_uuid: "2387cfabf2d84795915ecb496461ee14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } } }
I20260925 01:32:42.957659 10836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2387cfabf2d84795915ecb496461ee14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2387cfabf2d84795915ecb496461ee14" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 40651 } } }
I20260925 01:32:42.958037 10837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:42.958079 10836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2387cfabf2d84795915ecb496461ee14 [sys.catalog]: This master's current role is: LEADER
I20260925 01:32:42.971115 10842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:32:42.983023 10842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:32:42.985386 10842 catalog_manager.cc:1259] Loaded cluster ID: 286a907ab3954d24bf765f3be254655e
I20260925 01:32:42.985692 10842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 01:32:42.988778 10842 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f66d53ab9ad google::LogMessage::Fail() at ??:0
@ 0x7f66d53ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7f66d53ab4de google::LogMessage::Flush() at ??:0
@ 0x7f66d53aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f66eadde512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f66eaddb976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f66eae63610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f66eae63439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f66d7ff7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f66d7ffa91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f66d7ffa799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f66d7fd5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f66db2af6db start_thread at ??:0
@ 0x7f66d424561f clone at ??:0
W20260925 01:32:43.313987 10833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:43.314378 10833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:43.314579 10833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:43.322247 10833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:43.322400 10833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:43.322592 10833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:32:43.336309 10833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:33579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=44945
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:43.338162 10833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:43.339937 10833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:43.361699 10872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:43.361742 10875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:43.361811 10873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:43.363709 10833 server_base.cc:1034] running on GCE node
I20260925 01:32:43.364645 10833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:43.366173 10833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:43.367441 10833 hybrid_clock.cc:648] HybridClock initialized: now 1790299963367362 us; error 72 us; skew 500 ppm
I20260925 01:32:43.367885 10833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:43 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299963, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:43.372653 10833 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:32:43.374961 10833 webserver.cc:460] Webserver started at http://127.27.160.1:44945/ using document root <none> and password file <none>
I20260925 01:32:43.376060 10833 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:43.376235 10833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:43.385416 10833 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 01:32:43.388592 10882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:43.390399 10833 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:32:43.390573 10833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "faed72eb6aff47da88f59700c3c3666b"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1f0605ff5d6262d29c6ee0250c74207e"
server_key_iv: "33c25efc564ce326605587391e8cbbf8"
server_key_version: "encryptionkey@0"
I20260925 01:32:43.391613 10833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:43.407923 10833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:43.412643 10833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:43.412922 10833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:43.414109 10833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:43.415752 10833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:43.415851 10833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:43.415941 10833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:43.415999 10833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:43.453193 10833 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:33579
I20260925 01:32:43.453249 10994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:33579 every 8 connection(s)
I20260925 01:32:43.454785 10833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:32:43.459566 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 10833
I20260925 01:32:43.460428 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:46021
--local_ip_for_outbound_sockets=127.27.160.2
--tserver_master_addrs=127.27.160.62:40651
--webserver_port=39567
--webserver_interface=127.27.160.2
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:32:43.460541 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:43.458091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:57531)
0925 01:32:43.458848 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:43.459185 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":287,"threads_started":1}
W20260925 01:32:43.461388 10995 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:40651 (0 consecutive failures): Network error: Failed to ping master at 127.27.160.62:40651: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
I20260925 01:32:43.462483 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:43.462003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:41053)
0925 01:32:43.462219 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:43.462322 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 01:32:43.463572 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:43.463112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:34929)
0925 01:32:43.463303 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:43.463407 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 01:32:43.463794 10995 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:32:43.815083 10998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:43.815505 10998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:43.815657 10998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:43.823346 10998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:43.823486 10998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:43.823670 10998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:32:43.833945 10998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:46021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=39567
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:43.835395 10998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:43.836858 10998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:43.851068 11004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:43.851017 11003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:43.851322 11006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:43.852375 10998 server_base.cc:1034] running on GCE node
I20260925 01:32:43.853109 10998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:43.854234 10998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:43.855449 10998 hybrid_clock.cc:648] HybridClock initialized: now 1790299963855375 us; error 66 us; skew 500 ppm
I20260925 01:32:43.855794 10998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:43 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299963, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:43.859305 10998 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:32:43.860988 10998 webserver.cc:460] Webserver started at http://127.27.160.2:39567/ using document root <none> and password file <none>
I20260925 01:32:43.861793 10998 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:43.861960 10998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:43.868016 10998 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 01:32:43.871286 11013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:43.872848 10998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:32:43.873061 10998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "582e38dd0b584d5a8f4f8bf79ad00c1e"
format_stamp: "Formatted at 2026-09-25 01:32:37 on dist-test-slave-h6hf"
server_key: "1feece858f52b0609bdd2b37cd9a2ce3"
server_key_iv: "736a2052663b539dbe3eb6cc97e145f2"
server_key_version: "encryptionkey@0"
I20260925 01:32:43.874239 10998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:43.888013 10998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:43.892663 10998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:43.892988 10998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:43.894132 10998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:43.895645 10998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:43.895782 10998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:43.895919 10998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:43.896006 10998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:43.931955 10998 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:46021
I20260925 01:32:43.932009 11125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:46021 every 8 connection(s)
I20260925 01:32:43.933596 10998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:32:43.937711 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 10998
I20260925 01:32:43.938434 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:37535
--local_ip_for_outbound_sockets=127.27.160.3
--tserver_master_addrs=127.27.160.62:40651
--webserver_port=42139
--webserver_interface=127.27.160.3
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:32:43.939348 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:43.936974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:46393)
0925 01:32:43.937953 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:43.938249 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":436,"threads_started":1}
W20260925 01:32:43.940122 11126 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:40651 (0 consecutive failures): Network error: Failed to ping master at 127.27.160.62:40651: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
I20260925 01:32:43.941236 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:43.940779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:33839)
0925 01:32:43.940946 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:43.941019 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 01:32:43.942338 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:43.941932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:41807)
0925 01:32:43.942104 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:43.942194 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 01:32:43.942616 11126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:32:44.283973 11129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:32:44.284327 11129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:32:44.284505 11129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:32:44.291914 11129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:32:44.292109 11129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:32:44.292312 11129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:32:44.302675 11129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:37535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=42139
--tserver_master_addrs=127.27.160.62:40651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:32:44.304199 11129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:32:44.305621 11129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:32:44.319453 11137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:44.319473 11135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:32:44.319589 11134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:32:44.320372 11129 server_base.cc:1034] running on GCE node
I20260925 01:32:44.321075 11129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:32:44.322242 11129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:32:44.323482 11129 hybrid_clock.cc:648] HybridClock initialized: now 1790299964323419 us; error 59 us; skew 500 ppm
I20260925 01:32:44.323813 11129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:32:44 dist-test-slave-h6hf krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299964, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:32:44.327034 11129 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:32:44.328589 11129 webserver.cc:460] Webserver started at http://127.27.160.3:42139/ using document root <none> and password file <none>
I20260925 01:32:44.329346 11129 fs_manager.cc:359] Metadata directory not provided
I20260925 01:32:44.329458 11129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:32:44.334909 11129 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:32:44.337815 11144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:44.339282 11129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:32:44.339428 11129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "054e01f6aa2347c890e8f4e834006bef"
format_stamp: "Formatted at 2026-09-25 01:32:38 on dist-test-slave-h6hf"
server_key: "1709202b120096b14f5079b7eaeaeb80"
server_key_iv: "fe84fcffaf22cae1b5bf94f335e7a7b9"
server_key_version: "encryptionkey@0"
I20260925 01:32:44.340435 11129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:32:44.362411 11129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:32:44.367424 11129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:32:44.367703 11129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:32:44.368782 11129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:32:44.370213 11129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:32:44.370318 11129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:44.370482 11129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:32:44.370594 11129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:32:44.408350 11129 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:37535
I20260925 01:32:44.408402 11256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:37535 every 8 connection(s)
I20260925 01:32:44.409977 11129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:32:44.415127 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:44.413065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:47241)
0925 01:32:44.413955 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:44.414199 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":316,"threads_started":1}
I20260925 01:32:44.415808 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 11129
W20260925 01:32:44.416077 11257 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:40651 (0 consecutive failures): Network error: Failed to ping master at 127.27.160.62:40651: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
I20260925 01:32:44.417273 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:44.416802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:58595)
0925 01:32:44.417006 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:44.417091 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 01:32:44.418295 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:44.417959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:41061)
0925 01:32:44.418098 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:44.418177 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 01:32:44.418556 11257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:32:44.465422 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:44.464392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:35863)
0925 01:32:44.464917 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:44.465042 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":252,"threads_started":1}
I20260925 01:32:44.944320 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:44.943334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:58383)
0925 01:32:44.943912 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:44.944035 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":337,"threads_started":1}
I20260925 01:32:45.420385 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:45.419364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:49185)
0925 01:32:45.420002 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:45.420126 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":299,"threads_started":1}
I20260925 01:32:45.467283 11264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:45.466444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:39597)
0925 01:32:45.466998 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:45.467094 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":261,"threads_started":1}
I20260925 01:32:45.946457 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:45.945580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:60105)
0925 01:32:45.946151 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:45.946276 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":315,"threads_started":1}
I20260925 01:32:46.422497 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:46.421595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:48827)
0925 01:32:46.422201 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:46.422294 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":336,"threads_started":1}
I20260925 01:32:46.469285 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:46.468444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:53377)
0925 01:32:46.468959 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:46.469037 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":305,"threads_started":1}
I20260925 01:32:46.948247 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:46.947517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:47219)
0925 01:32:46.947990 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:46.948075 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260925 01:32:47.424556 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:47.423669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:57575)
0925 01:32:47.424204 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:47.424311 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":265,"threads_started":1}
I20260925 01:32:47.471421 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:47.470494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:50135)
0925 01:32:47.471109 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:47.471216 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":332,"threads_started":1}
I20260925 01:32:47.950157 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:47.949345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:42635)
0925 01:32:47.949833 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:47.949925 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":216,"threads_started":1}
I20260925 01:32:48.426473 11272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:48.425669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:56263)
0925 01:32:48.426202 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:48.426289 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":308,"threads_started":1}
I20260925 01:32:48.473417 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:48.472519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:53557)
0925 01:32:48.473056 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:48.473223 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":280,"threads_started":1}
I20260925 01:32:48.951894 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:48.951151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:40983)
0925 01:32:48.951623 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:48.951717 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
I20260925 01:32:49.428647 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:49.427621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:41899)
0925 01:32:49.428296 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:49.428441 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":288,"threads_started":1}
I20260925 01:32:49.475531 11276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:49.474574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:35091)
0925 01:32:49.475245 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:49.475361 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":374,"threads_started":1}
I20260925 01:32:49.953873 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:49.952957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:45121)
0925 01:32:49.953599 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:49.953701 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":386,"threads_started":1}
I20260925 01:32:50.430595 11278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:50.429767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:57269)
0925 01:32:50.430315 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:50.430422 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":280,"threads_started":1}
I20260925 01:32:50.477469 11279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:50.476508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:39419)
0925 01:32:50.477061 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:50.477244 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":240,"threads_started":1}
I20260925 01:32:50.955762 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:50.954928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:50353)
0925 01:32:50.955448 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:50.955541 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":231,"threads_started":1}
I20260925 01:32:51.432482 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:51.431649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:60997)
0925 01:32:51.432174 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:51.432264 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":234,"threads_started":1}
I20260925 01:32:51.479480 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:51.478681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:60525)
0925 01:32:51.479184 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:51.479272 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":266,"threads_started":1}
I20260925 01:32:51.957763 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:51.956844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:58147)
0925 01:32:51.957428 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:51.957529 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":321,"threads_started":1}
I20260925 01:32:52.434805 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:52.433850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:35813)
0925 01:32:52.434499 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:52.434618 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":359,"threads_started":1}
I20260925 01:32:52.481422 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:52.480433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:50219)
0925 01:32:52.481085 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:52.481239 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":434,"threads_started":1}
I20260925 01:32:52.959851 11286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:52.958978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:44821)
0925 01:32:52.959576 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:52.959671 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":281,"threads_started":1}
I20260925 01:32:53.437040 11287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:53.436048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:50029)
0925 01:32:53.436739 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:53.436846 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":390,"threads_started":1}
I20260925 01:32:53.483430 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:53.482468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:43427)
0925 01:32:53.483125 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:53.483231 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":408,"threads_started":1}
I20260925 01:32:53.961908 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:53.961010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:42809)
0925 01:32:53.961622 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:53.961733 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":380,"threads_started":1}
I20260925 01:32:54.438884 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:54.438134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:59545)
0925 01:32:54.438627 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:54.438718 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":282,"threads_started":1}
I20260925 01:32:54.485438 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:54.484565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:59145)
0925 01:32:54.485051 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:54.485239 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":245,"threads_started":1}
I20260925 01:32:54.963999 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:54.963145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:51801)
0925 01:32:54.963653 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:54.963805 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":251,"threads_started":1}
I20260925 01:32:55.441098 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:55.440181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:46243)
0925 01:32:55.440801 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:55.440921 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":311,"threads_started":1}
I20260925 01:32:55.487172 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:55.486424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:56229)
0925 01:32:55.486931 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:55.487012 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":288,"threads_started":1}
I20260925 01:32:55.965821 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:55.965013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:34077)
0925 01:32:55.965571 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:55.965652 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":356,"threads_started":1}
I20260925 01:32:56.443040 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:56.442220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:46651)
0925 01:32:56.442768 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:56.442857 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":295,"threads_started":1}
I20260925 01:32:56.488978 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:56.488136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:47577)
0925 01:32:56.488652 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:56.488774 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":250,"threads_started":1}
I20260925 01:32:56.967681 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:56.966841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:50425)
0925 01:32:56.967389 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:56.967489 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":258,"threads_started":1}
I20260925 01:32:57.445398 11299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:57.444162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:46451)
0925 01:32:57.444943 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:57.445125 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":422,"threads_started":1}
I20260925 01:32:57.490685 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:57.489912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:40325)
0925 01:32:57.490397 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:57.490499 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":248,"threads_started":1}
I20260925 01:32:57.969645 11301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:57.968800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:49867)
0925 01:32:57.969361 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:57.969455 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":246,"threads_started":1}
I20260925 01:32:58.447139 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:58.446332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:49545)
0925 01:32:58.446852 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:58.446956 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":247,"threads_started":1}
I20260925 01:32:58.492731 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:58.491970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:58621)
0925 01:32:58.492451 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:58.492547 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":228,"threads_started":1}
I20260925 01:32:58.971685 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:58.970874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.2:59095)
0925 01:32:58.971359 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:58.971472 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":246,"threads_started":1}
I20260925 01:32:59.418442 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 10833
I20260925 01:32:59.442489 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:59.419843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.27.160.1:33579)
0925 01:32:59.420576 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:59.420597 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:59.420644 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:32:59.423213 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:59.423224 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:59.423748 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:59.424089 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:59.425594 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:59.426973 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:59.428222 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:59.428944 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:59.429083 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:59.435882 (+ 6799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:59.435892 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:59.435910 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:59.435966 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:59.438914 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:59.439638 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:59.439646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:59.439653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:59.439883 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:59.440247 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:59.440259 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:59.440261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:59.440893 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:59.441262 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:59.441634 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 01:32:59.442175 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":368,"threads_started":1}
I20260925 01:32:59.444049 10990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:59.449672 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:59.448489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.3:42393)
0925 01:32:59.449373 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:59.449487 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":618,"threads_started":1}
I20260925 01:32:59.534791 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 10833
I20260925 01:32:59.534902 11308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:59.532613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:40651 (local address 127.27.160.1:47381)
0925 01:32:59.534039 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0925 01:32:59.534514 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:40651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1272,"thread_start_us":1090,"threads_started":1}
I20260925 01:32:59.554731 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 10998
I20260925 01:32:59.574890 11309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:59.555959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.27.160.2:46021)
0925 01:32:59.556540 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:59.556555 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:59.556597 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:32:59.558376 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:59.558388 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:59.558889 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:59.559235 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:59.560788 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:59.562343 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:59.563764 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:59.564567 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:59.564745 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:59.567509 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:59.567518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:59.567538 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:59.567589 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:59.571096 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:59.571917 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:59.571926 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:59.571935 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:59.572144 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:59.572517 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:59.572521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:59.572523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:59.573373 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:59.573640 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:59.574078 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 01:32:59.574537 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":281,"threads_started":1}
I20260925 01:32:59.576318 11121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:59.660930 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 10998
I20260925 01:32:59.679675 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 11129
I20260925 01:32:59.698489 11311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:32:59.680885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.27.160.3:37535)
0925 01:32:59.681603 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 01:32:59.681617 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:32:59.681658 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:32:59.683255 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:32:59.683264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:32:59.683698 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:32:59.683991 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:32:59.685640 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:59.686911 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:59.688330 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:32:59.688985 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:32:59.689175 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:32:59.691988 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:32:59.691997 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:32:59.692014 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:32:59.692054 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:32:59.695118 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:59.695834 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:59.695841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:59.695848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:59.696014 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:32:59.696400 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:32:59.696403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:32:59.696405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:32:59.696944 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:32:59.697248 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 01:32:59.697727 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 01:32:59.698167 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":430,"threads_started":1}
I20260925 01:32:59.700045 11252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:32:59.784119 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 11129
2026-09-25T01:32:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25174 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:32:59 dist-test-slave-h6hf krb5kdc[11315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:32:59 dist-test-slave-h6hf krb5kdc[11315](info): set up 2 sockets
Sep 25 01:32:59 dist-test-slave-h6hf krb5kdc[11315](info): commencing operation
krb5kdc: starting...
W20260925 01:33:00.938978 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:33:01 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:33:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:33:01.195066 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37989
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:37989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:33:01.571514 11328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:01.571835 11328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:01.571928 11328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:01.579355 11328 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:33:01.579463 11328 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:33:01.579502 11328 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:33:01.579547 11328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:33:01.579602 11328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:01.579665 11328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:01.579710 11328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:33:01.579759 11328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:33:01.589871 11328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:37989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:01.591157 11328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:01.592417 11328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:01.604426 11333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:01.604530 11334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:01.604561 11336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:01.605535 11328 server_base.cc:1034] running on GCE node
I20260925 01:33:01.606369 11328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:01.607846 11328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:01.609081 11328 hybrid_clock.cc:648] HybridClock initialized: now 1790299981609015 us; error 51 us; skew 500 ppm
I20260925 01:33:01.609486 11328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:01 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299981, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:01.612691 11328 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:33:01.614363 11328 webserver.cc:460] Webserver started at http://127.27.160.62:34457/ using document root <none> and password file <none>
I20260925 01:33:01.615105 11328 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:01.615238 11328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:01.615568 11328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:33:01.618701 11328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "10e9c376b8034b5e89947df54c47c767"
format_stamp: "Formatted at 2026-09-25 01:33:01 on dist-test-slave-h6hf"
server_key: "5a3618e954545b886780693dcd217ae7"
server_key_iv: "e324e5b1e4f2097ce83edc642d08eaa8"
server_key_version: "encryptionkey@0"
I20260925 01:33:01.619536 11328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "10e9c376b8034b5e89947df54c47c767"
format_stamp: "Formatted at 2026-09-25 01:33:01 on dist-test-slave-h6hf"
server_key: "5a3618e954545b886780693dcd217ae7"
server_key_iv: "e324e5b1e4f2097ce83edc642d08eaa8"
server_key_version: "encryptionkey@0"
I20260925 01:33:01.625010 11328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:33:01.628767 11343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:01.630283 11328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:33:01.630492 11328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "10e9c376b8034b5e89947df54c47c767"
format_stamp: "Formatted at 2026-09-25 01:33:01 on dist-test-slave-h6hf"
server_key: "5a3618e954545b886780693dcd217ae7"
server_key_iv: "e324e5b1e4f2097ce83edc642d08eaa8"
server_key_version: "encryptionkey@0"
I20260925 01:33:01.630739 11328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:01.647908 11328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:01.652699 11328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:01.652989 11328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:01.673000 11328 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:37989
I20260925 01:33:01.673023 11394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:37989 every 8 connection(s)
I20260925 01:33:01.674448 11328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:33:01.678637 11395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:33:01.681648 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 11328
I20260925 01:33:01.681885 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:33:01.682575 28288 external_mini_cluster.cc:1428] Setting key 701c32c37e7e71a24daa4317e70b50cd
I20260925 01:33:01.692121 11395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Bootstrap starting.
I20260925 01:33:01.696508 11395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Neither blocks nor log segments found. Creating new log.
I20260925 01:33:01.697855 11395 log.cc:824] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:33:01 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:33:01.701107 11395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: No bootstrap required, opened a new log
I20260925 01:33:01.707540 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:01.684716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.27.160.62:37989)
0925 01:33:01.685525 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:01.685537 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:01.685577 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:33:01.687286 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:01.687307 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:01.687708 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:01.688041 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:01.690172 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:01.691485 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:01.693676 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:01.694381 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:01.694523 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:01.699864 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:01.699872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:01.699888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:01.699923 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:01.703147 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:01.703916 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:01.703923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:01.703930 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:01.704100 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:01.704530 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:01.704534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:01.704536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:01.705225 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:01.705556 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:01.705988 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 01:33:01.706466 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":391,"threads_started":1}
I20260925 01:33:01.710217 11395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:01.710713 11395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:33:01.710811 11395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e9c376b8034b5e89947df54c47c767, State: Initialized, Role: FOLLOWER
I20260925 01:33:01.711346 11395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:01.711573 11395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:33:01.711676 11395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:33:01.711817 11395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:33:01.714715 11395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:01.715406 11395 leader_election.cc:302] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10e9c376b8034b5e89947df54c47c767; no voters:
I20260925 01:33:01.716096 11395 leader_election.cc:288] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:33:01.716341 11400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:33:01.717239 11400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 1 LEADER]: Becoming Leader. State: Replica: 10e9c376b8034b5e89947df54c47c767, State: Running, Role: LEADER
I20260925 01:33:01.718034 11400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:01.718837 11395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:33:01.721513 11402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10e9c376b8034b5e89947df54c47c767. Latest consensus state: current_term: 1 leader_uuid: "10e9c376b8034b5e89947df54c47c767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } } }
I20260925 01:33:01.721587 11401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10e9c376b8034b5e89947df54c47c767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } } }
I20260925 01:33:01.722138 11402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: This master's current role is: LEADER
I20260925 01:33:01.723353 11401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: This master's current role is: LEADER
I20260925 01:33:01.724285 11409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:33:01.733156 11409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:33:01.744040 11409 catalog_manager.cc:1347] Generated new cluster ID: 4e6fb7c2483e4ce1bb30e8658a453539
I20260925 01:33:01.744197 11409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:33:01.760118 11409 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:33:01.761705 11409 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:33:01.768008 11409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Generated new TSK 0
I20260925 01:33:01.768930 11409 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:33:01.837893 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37989
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:33:02.187506 11421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:02.187831 11421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:02.187996 11421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:02.195683 11421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:02.195816 11421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:02.196005 11421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:33:02.206692 11421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:02.208105 11421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:02.209652 11421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:02.223908 11429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:02.224279 11427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:02.224769 11426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:02.225133 11421 server_base.cc:1034] running on GCE node
I20260925 01:33:02.225970 11421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:02.227092 11421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:02.228355 11421 hybrid_clock.cc:648] HybridClock initialized: now 1790299982228291 us; error 44 us; skew 500 ppm
I20260925 01:33:02.228832 11421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:02 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299982, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:02.232834 11421 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:33:02.234908 11421 webserver.cc:460] Webserver started at http://127.27.160.1:36601/ using document root <none> and password file <none>
I20260925 01:33:02.235842 11421 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:02.236008 11421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:02.236368 11421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:33:02.239905 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "b83f7a4f20d645b08667e14c09c2bb52"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "b9dfdf23c382dc0459cf3d33d74abcce"
server_key_iv: "dcac05e2bc97d61c98c4aa715da45592"
server_key_version: "encryptionkey@0"
I20260925 01:33:02.240823 11421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "b83f7a4f20d645b08667e14c09c2bb52"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "b9dfdf23c382dc0459cf3d33d74abcce"
server_key_iv: "dcac05e2bc97d61c98c4aa715da45592"
server_key_version: "encryptionkey@0"
I20260925 01:33:02.247072 11421 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260925 01:33:02.251449 11436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:02.253443 11421 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:33:02.253660 11421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "b83f7a4f20d645b08667e14c09c2bb52"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "b9dfdf23c382dc0459cf3d33d74abcce"
server_key_iv: "dcac05e2bc97d61c98c4aa715da45592"
server_key_version: "encryptionkey@0"
I20260925 01:33:02.253913 11421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:02.269747 11421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:02.274294 11421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:02.274618 11421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:02.275676 11421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:02.277174 11421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:33:02.277287 11421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:02.277437 11421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:33:02.277515 11421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:02.311929 11421 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:40009
I20260925 01:33:02.311977 11548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:40009 every 8 connection(s)
I20260925 01:33:02.313437 11421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:33:02.314859 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 11421
I20260925 01:33:02.315093 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:33:02.315939 28288 external_mini_cluster.cc:1428] Setting key 93f5f509e9a8f62e73e51719fd6096e4
Sep 25 01:33:02 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299982, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:33:02.341008 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:02.316637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:39127 (local address 127.27.160.62:37989)
0925 01:33:02.317100 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:02.317104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:02.318991 (+ 1887us) server_negotiation.cc:407] Connection header received
0925 01:33:02.321182 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:02.321190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:02.321306 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:02.321434 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:02.323835 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:02.325094 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:02.327917 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:02.328377 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:02.328503 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:02.333740 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:02.333747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:02.333751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:02.333791 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:02.336935 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:02.338104 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:02.338109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:02.338112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:02.338260 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:02.338877 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:02.338882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:02.338884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:02.339194 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:02.339334 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:02.340465 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0925 01:33:02.340672 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260925 01:33:02.342677 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:02.317452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:39127)
0925 01:33:02.318459 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:02.318514 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:02.320818 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:02.321742 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:02.321758 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:02.322700 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:33:02.323605 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:02.323634 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:02.325364 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:02.325373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:02.327714 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:02.327738 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:02.328650 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:02.328659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:02.329031 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:02.329987 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:33:02.330025 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:33:02.333411 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:33:02.337155 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:02.337170 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:02.337193 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:02.337949 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:02.338399 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:02.338407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:02.338414 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:02.338721 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:02.339351 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:33:02.339361 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:33:02.339936 (+ 575us) client_negotiation.cc:769] Sending connection context
0925 01:33:02.340313 (+ 377us) client_negotiation.cc:240] Negotiation successful
0925 01:33:02.341001 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":437,"threads_started":1}
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:33:02.344638 11549 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37989
I20260925 01:33:02.345011 11549 heartbeater.cc:464] Registering TS with master...
I20260925 01:33:02.345880 11549 heartbeater.cc:511] Master 127.27.160.62:37989 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
I20260925 01:33:02.348727 11360 ts_manager.cc:194] Registered new tserver with Master: b83f7a4f20d645b08667e14c09c2bb52 (127.27.160.1:40009)
I20260925 01:33:02.351075 11360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:39127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:33:02.379993 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37989
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:33:02.731321 11555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:02.731637 11555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:02.731748 11555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:02.739466 11555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:02.739596 11555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:02.739722 11555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:33:02.749953 11555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:02.751260 11555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:02.752525 11555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:02.767357 11560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:02.767436 11561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:02.767506 11563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:02.768260 11555 server_base.cc:1034] running on GCE node
I20260925 01:33:02.768999 11555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:02.770124 11555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:02.771391 11555 hybrid_clock.cc:648] HybridClock initialized: now 1790299982771337 us; error 52 us; skew 500 ppm
I20260925 01:33:02.771782 11555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:02 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299982, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:02.775283 11555 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:33:02.776938 11555 webserver.cc:460] Webserver started at http://127.27.160.2:44885/ using document root <none> and password file <none>
I20260925 01:33:02.777729 11555 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:02.777884 11555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:02.778205 11555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:33:02.781371 11555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "baa5e7ab94744b32a03f2877a00a2323"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "0f3f83a9982cad11fe7406641c83271b"
server_key_iv: "9ac7014cf12c3b2e1b58e7b79a35c333"
server_key_version: "encryptionkey@0"
I20260925 01:33:02.782194 11555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "baa5e7ab94744b32a03f2877a00a2323"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "0f3f83a9982cad11fe7406641c83271b"
server_key_iv: "9ac7014cf12c3b2e1b58e7b79a35c333"
server_key_version: "encryptionkey@0"
I20260925 01:33:02.787894 11555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260925 01:33:02.791906 11570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:02.793494 11555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:33:02.793682 11555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "baa5e7ab94744b32a03f2877a00a2323"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "0f3f83a9982cad11fe7406641c83271b"
server_key_iv: "9ac7014cf12c3b2e1b58e7b79a35c333"
server_key_version: "encryptionkey@0"
I20260925 01:33:02.793908 11555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:02.809432 11555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:02.815227 11555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:02.815533 11555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:02.816534 11555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:02.818058 11555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:33:02.818157 11555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:02.818291 11555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:33:02.818362 11555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:02.863137 11555 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:41969
I20260925 01:33:02.863154 11682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:41969 every 8 connection(s)
I20260925 01:33:02.864579 11555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:33:02.865864 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 11555
I20260925 01:33:02.866077 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:33:02.866559 28288 external_mini_cluster.cc:1428] Setting key 2515a983b206873bd45e2c4e36a90d31
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Sep 25 01:33:02 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299982, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:33:02.912876 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:02.868554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:34615 (local address 127.27.160.62:37989)
0925 01:33:02.881554 (+ 13000us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:02.881562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:02.882700 (+ 1138us) server_negotiation.cc:407] Connection header received
0925 01:33:02.886158 (+ 3458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:02.886167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:02.886277 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:02.886417 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:02.889216 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:02.890576 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:02.893810 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:02.894606 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:02.894770 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:02.903801 (+ 9031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:02.903811 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:02.903819 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:02.903878 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:02.908113 (+ 4235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:02.909471 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:02.909481 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:02.909487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:02.909673 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:02.910280 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:02.910287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:02.910291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:02.910729 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:02.910922 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:02.912035 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0925 01:33:02.912328 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12830,"thread_start_us":13038,"threads_started":1}
I20260925 01:33:02.914531 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:02.869282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:34615)
0925 01:33:02.882450 (+ 13168us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:02.882519 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:02.885725 (+ 3206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:02.886828 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:02.886845 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:02.887864 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:33:02.888903 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:02.888944 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:02.890808 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:02.890816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:02.893560 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:02.893586 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:02.894944 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:02.894955 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:02.895365 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:02.896357 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:33:02.896397 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:33:02.903403 (+ 7006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:33:02.908386 (+ 4983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:02.908403 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:02.908434 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:02.909296 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:02.909803 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:02.909810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:02.909815 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:02.910117 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:02.910885 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:33:02.910896 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:33:02.911483 (+ 587us) client_negotiation.cc:769] Sending connection context
0925 01:33:02.911880 (+ 397us) client_negotiation.cc:240] Negotiation successful
0925 01:33:02.912630 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12768,"thread_start_us":12520,"threads_started":1}
I20260925 01:33:02.916460 11683 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37989
I20260925 01:33:02.916823 11683 heartbeater.cc:464] Registering TS with master...
I20260925 01:33:02.917840 11683 heartbeater.cc:511] Master 127.27.160.62:37989 requested a full tablet report, sending...
I20260925 01:33:02.919811 11360 ts_manager.cc:194] Registered new tserver with Master: baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969)
I20260925 01:33:02.921304 11360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:34615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:33:02.937252 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37989
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:33:03.282532 11689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:03.282846 11689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:03.283020 11689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:03.290431 11689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:03.290557 11689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:03.290738 11689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:33:03.302711 11689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:03.304095 11689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:03.305526 11689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:03.319072 11694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:03.319144 11695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:03.319166 11697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:03.320066 11689 server_base.cc:1034] running on GCE node
I20260925 01:33:03.320763 11689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:03.322062 11689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:03.323259 11689 hybrid_clock.cc:648] HybridClock initialized: now 1790299983323171 us; error 90 us; skew 500 ppm
I20260925 01:33:03.323647 11689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:03 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299983, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:03.333534 11689 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:33:03.335389 11689 webserver.cc:460] Webserver started at http://127.27.160.3:45663/ using document root <none> and password file <none>
I20260925 01:33:03.336150 11689 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:03.336259 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:03.336606 11689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:33:03.339740 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "87909428a991490d80bd73479305f956"
format_stamp: "Formatted at 2026-09-25 01:33:03 on dist-test-slave-h6hf"
server_key: "ba993ea6dee43634bdd5dcc7b84135e5"
server_key_iv: "38d613830114ecd494e68508ed76ebca"
server_key_version: "encryptionkey@0"
I20260925 01:33:03.340598 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "87909428a991490d80bd73479305f956"
format_stamp: "Formatted at 2026-09-25 01:33:03 on dist-test-slave-h6hf"
server_key: "ba993ea6dee43634bdd5dcc7b84135e5"
server_key_iv: "38d613830114ecd494e68508ed76ebca"
server_key_version: "encryptionkey@0"
I20260925 01:33:03.346563 11689 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 01:33:03.350368 11704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:03.351799 11689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:33:03.352005 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "87909428a991490d80bd73479305f956"
format_stamp: "Formatted at 2026-09-25 01:33:03 on dist-test-slave-h6hf"
server_key: "ba993ea6dee43634bdd5dcc7b84135e5"
server_key_iv: "38d613830114ecd494e68508ed76ebca"
server_key_version: "encryptionkey@0"
I20260925 01:33:03.352245 11689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:03.354938 11549 heartbeater.cc:503] Master 127.27.160.62:37989 was elected leader, sending a full tablet report...
I20260925 01:33:03.370877 11689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:03.375591 11689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:03.375903 11689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:03.377012 11689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:03.378609 11689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:33:03.378719 11689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:03.378877 11689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:33:03.378948 11689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:03.419595 11689 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:33397
I20260925 01:33:03.419787 11816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:33397 every 8 connection(s)
I20260925 01:33:03.421332 11689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:33:03.424242 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 11689
I20260925 01:33:03.424439 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:33:03.425024 28288 external_mini_cluster.cc:1428] Setting key 90b3148cf4ce1c1e97fff6ed926b1fcf
Sep 25 01:33:03 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299983, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:33:03.446568 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.424185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:59363 (local address 127.27.160.62:37989)
0925 01:33:03.426937 (+ 2752us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.426945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.426967 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:33:03.429292 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.429296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.429383 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.429473 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:03.431209 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.432026 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.434125 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.434609 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.434706 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.438802 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:03.438807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:03.438810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:03.438865 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:03.442783 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:03.443868 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:03.443874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:03.443877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:03.444036 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:03.444570 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:03.444575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:03.444579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:03.444939 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:03.445109 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.445911 (+ 802us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.446150 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2623,"thread_start_us":2500,"threads_started":1}
I20260925 01:33:03.447790 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.424999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:59363)
0925 01:33:03.426318 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:03.426400 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:03.428964 (+ 2564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:03.429727 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:03.429736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:03.430383 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:33:03.431037 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.431058 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.432226 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.432231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.433987 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.434001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.435026 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.435038 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.435297 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.436054 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:33:03.436080 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:33:03.438586 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:33:03.442990 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:03.443039 (+ 49us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:03.443060 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:03.443695 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:03.444151 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:03.444159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:03.444164 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:03.444442 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:03.445076 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:33:03.445085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:33:03.445522 (+ 437us) client_negotiation.cc:769] Sending connection context
0925 01:33:03.445808 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 01:33:03.446395 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":512,"threads_started":1}
I20260925 01:33:03.449747 11817 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37989
I20260925 01:33:03.450116 11817 heartbeater.cc:464] Registering TS with master...
I20260925 01:33:03.451114 11817 heartbeater.cc:511] Master 127.27.160.62:37989 requested a full tablet report, sending...
I20260925 01:33:03.453104 11360 ts_manager.cc:194] Registered new tserver with Master: 87909428a991490d80bd73479305f956 (127.27.160.3:33397)
I20260925 01:33:03.454370 11360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:59363
I20260925 01:33:03.455655 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:33:03.483003 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.462577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.27.160.62:37989)
0925 01:33:03.462795 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.462801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.463193 (+ 392us) server_negotiation.cc:407] Connection header received
0925 01:33:03.463476 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.463482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.463592 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.463746 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:03.466024 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.467433 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.469754 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.470275 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.470435 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.474493 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:03.474504 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:03.474511 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:03.474568 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:03.478691 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:03.479776 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:03.479784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:03.479791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:03.479996 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:03.480500 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:03.480508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:03.480514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:03.481069 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:03.481398 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.482106 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.482380 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 01:33:03.488080 11360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:33:03.499238 11360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:33:03.523847 11833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.511815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:33397 (local address 127.0.0.1:48672)
0925 01:33:03.513691 (+ 1876us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:03.513714 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:03.513835 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:03.515673 (+ 1838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:03.515683 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:03.515721 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:03.516265 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.516279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.518195 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.518203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.520936 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.520953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.522646 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.522653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.522857 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.522959 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 01:33:03.523113 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 01:33:03.523383 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1728,"mutex_wait_us":93,"thread_start_us":3504,"threads_started":1}
I20260925 01:33:03.524999 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.512920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.27.160.3:33397)
0925 01:33:03.514918 (+ 1998us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.514928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.514973 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:33:03.515085 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.515094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.515397 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.516511 (+ 1114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:03.516794 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.518044 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.521146 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.522418 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.522530 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.522774 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.524374 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.524630 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1790,"thread_start_us":1623,"threads_started":1}
I20260925 01:33:03.525617 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.511993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.27.160.1:40009)
0925 01:33:03.515381 (+ 3388us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.515391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.515422 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:33:03.515523 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.515531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.515787 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.516724 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:03.516928 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.518482 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.522910 (+ 4428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.524396 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.524556 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.524805 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.525053 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.525261 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3164,"thread_start_us":2987,"threads_started":1}
I20260925 01:33:03.525740 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.512718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.27.160.2:41969)
0925 01:33:03.514313 (+ 1595us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.514326 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.515953 (+ 1627us) server_negotiation.cc:407] Connection header received
0925 01:33:03.516059 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.516068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.516290 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.516482 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:03.517391 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.518918 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.521884 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.523477 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.523628 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.525211 (+ 1583us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.525315 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.525485 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1433,"thread_start_us":1340,"threads_started":1}
I20260925 01:33:03.525055 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.512521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41969 (local address 127.0.0.1:32812)
0925 01:33:03.515644 (+ 3123us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:03.515662 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:03.515793 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:03.516728 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:03.516734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:03.516762 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:03.517179 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.517191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.519135 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.519140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.521692 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.521707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.524209 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.524216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.524389 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.524422 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:33:03.524493 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 01:33:03.524636 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2982,"mutex_wait_us":82,"thread_start_us":863,"threads_started":1}
I20260925 01:33:03.526332 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.511645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:60364)
0925 01:33:03.512782 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:03.512826 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:03.513039 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:03.516015 (+ 2976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:03.516019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:03.516044 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:03.516546 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.516560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.518641 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.518647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.520936 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.520953 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.524641 (+ 3688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.524647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.524851 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.524882 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:33:03.524989 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 01:33:03.525169 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":55,"thread_start_us":653,"threads_started":1}
I20260925 01:33:03.529065 11752 tablet_service.cc:1467] Processing CreateTablet for tablet 9baf5fc757e04ba38a458f93c971599e (DEFAULT_TABLE table=test-table [id=07d3832e6bc44cdca1f73f760e793f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:33:03.529695 11484 tablet_service.cc:1467] Processing CreateTablet for tablet 9baf5fc757e04ba38a458f93c971599e (DEFAULT_TABLE table=test-table [id=07d3832e6bc44cdca1f73f760e793f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:33:03.530804 11752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9baf5fc757e04ba38a458f93c971599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:33:03.531363 11484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9baf5fc757e04ba38a458f93c971599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:33:03.531467 11618 tablet_service.cc:1467] Processing CreateTablet for tablet 9baf5fc757e04ba38a458f93c971599e (DEFAULT_TABLE table=test-table [id=07d3832e6bc44cdca1f73f760e793f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:33:03.533679 11618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9baf5fc757e04ba38a458f93c971599e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:33:03.545171 11836 tablet_bootstrap.cc:492] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Bootstrap starting.
I20260925 01:33:03.548822 11836 tablet_bootstrap.cc:654] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Neither blocks nor log segments found. Creating new log.
I20260925 01:33:03.550894 11836 log.cc:824] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:03.553880 11839 tablet_bootstrap.cc:492] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Bootstrap starting.
I20260925 01:33:03.554374 11837 tablet_bootstrap.cc:492] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Bootstrap starting.
I20260925 01:33:03.554528 11836 tablet_bootstrap.cc:492] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: No bootstrap required, opened a new log
I20260925 01:33:03.554895 11836 ts_tablet_manager.cc:1397] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260925 01:33:03.557917 11839 tablet_bootstrap.cc:654] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Neither blocks nor log segments found. Creating new log.
I20260925 01:33:03.558001 11837 tablet_bootstrap.cc:654] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Neither blocks nor log segments found. Creating new log.
I20260925 01:33:03.559417 11837 log.cc:824] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:03.559389 11839 log.cc:824] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:03.562942 11839 tablet_bootstrap.cc:492] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: No bootstrap required, opened a new log
I20260925 01:33:03.563269 11839 ts_tablet_manager.cc:1397] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260925 01:33:03.563545 11837 tablet_bootstrap.cc:492] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: No bootstrap required, opened a new log
I20260925 01:33:03.563835 11837 ts_tablet_manager.cc:1397] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260925 01:33:03.565027 11836 raft_consensus.cc:348] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.565570 11836 raft_consensus.cc:374] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:33:03.565707 11836 raft_consensus.cc:729] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83f7a4f20d645b08667e14c09c2bb52, State: Initialized, Role: FOLLOWER
I20260925 01:33:03.566416 11836 consensus_queue.cc:260] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.571913 11836 ts_tablet_manager.cc:1428] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Time spent starting tablet: real 0.017s user 0.012s sys 0.000s
I20260925 01:33:03.574529 11837 raft_consensus.cc:348] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.574529 11839 raft_consensus.cc:348] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.575037 11839 raft_consensus.cc:374] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:33:03.575035 11837 raft_consensus.cc:374] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:33:03.575147 11839 raft_consensus.cc:729] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e7ab94744b32a03f2877a00a2323, State: Initialized, Role: FOLLOWER
I20260925 01:33:03.575147 11837 raft_consensus.cc:729] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87909428a991490d80bd73479305f956, State: Initialized, Role: FOLLOWER
I20260925 01:33:03.575824 11837 consensus_queue.cc:260] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.575824 11839 consensus_queue.cc:260] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.577059 11683 heartbeater.cc:503] Master 127.27.160.62:37989 was elected leader, sending a full tablet report...
I20260925 01:33:03.577582 11817 heartbeater.cc:503] Master 127.27.160.62:37989 was elected leader, sending a full tablet report...
I20260925 01:33:03.577966 11839 ts_tablet_manager.cc:1428] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260925 01:33:03.579468 11837 ts_tablet_manager.cc:1428] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
W20260925 01:33:03.617100 11684 tablet.cc:2369] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:33:03.673245 11818 tablet.cc:2369] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:33:03.816229 11550 tablet.cc:2369] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:33:03.845753 11844 raft_consensus.cc:482] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:33:03.846118 11844 raft_consensus.cc:504] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.847957 11844 leader_election.cc:288] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b83f7a4f20d645b08667e14c09c2bb52 (127.27.160.1:40009), baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969)
I20260925 01:33:03.854583 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.848247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41969 (local address 127.27.160.3:38887)
0925 01:33:03.848582 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:03.848602 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:03.848732 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:03.849191 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:03.849196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:03.849225 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:03.849444 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.849455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.850696 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.850702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.852568 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.852584 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.853952 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.853957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.854103 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.854128 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:33:03.854241 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:33:03.854353 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260925 01:33:03.854676 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.848664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:38887 (local address 127.27.160.2:41969)
0925 01:33:03.848890 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.848894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.848910 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:33:03.848970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.848973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.849031 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.849149 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:03.849611 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.850537 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.852753 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.853771 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.853953 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.854036 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.854271 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.854412 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:33:03.855753 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.848380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:35093 (local address 127.27.160.1:40009)
0925 01:33:03.848656 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.848665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.849460 (+ 795us) server_negotiation.cc:407] Connection header received
0925 01:33:03.849742 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.849748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.849825 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.849920 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:03.850389 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.851176 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.853552 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.854763 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.854924 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.855014 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.855374 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.855504 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:33:03.855785 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.848347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.27.160.3:35093)
0925 01:33:03.849331 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:03.849347 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:03.849514 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:03.849946 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:03.849949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:03.849970 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:03.850241 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.850250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.851330 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.851336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.853346 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:03.853360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.855074 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.855079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:03.855201 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.855224 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:33:03.855327 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 01:33:03.855457 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":558,"threads_started":1}
I20260925 01:33:03.855796 11638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9baf5fc757e04ba38a458f93c971599e" candidate_uuid: "87909428a991490d80bd73479305f956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7ab94744b32a03f2877a00a2323" is_pre_election: true
I20260925 01:33:03.856526 11638 raft_consensus.cc:2454] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87909428a991490d80bd73479305f956 in term 0.
I20260925 01:33:03.857424 11706 leader_election.cc:302] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87909428a991490d80bd73479305f956, baa5e7ab94744b32a03f2877a00a2323; no voters:
I20260925 01:33:03.857349 11504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9baf5fc757e04ba38a458f93c971599e" candidate_uuid: "87909428a991490d80bd73479305f956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83f7a4f20d645b08667e14c09c2bb52" is_pre_election: true
I20260925 01:33:03.857951 11844 raft_consensus.cc:2781] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:33:03.858134 11844 raft_consensus.cc:482] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:33:03.858062 11504 raft_consensus.cc:2454] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87909428a991490d80bd73479305f956 in term 0.
I20260925 01:33:03.858273 11844 raft_consensus.cc:3037] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:33:03.860940 11844 raft_consensus.cc:504] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.861771 11844 leader_election.cc:288] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [CANDIDATE]: Term 1 election: Requested vote from peers b83f7a4f20d645b08667e14c09c2bb52 (127.27.160.1:40009), baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969)
I20260925 01:33:03.862289 11504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9baf5fc757e04ba38a458f93c971599e" candidate_uuid: "87909428a991490d80bd73479305f956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83f7a4f20d645b08667e14c09c2bb52"
I20260925 01:33:03.862334 11638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9baf5fc757e04ba38a458f93c971599e" candidate_uuid: "87909428a991490d80bd73479305f956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7ab94744b32a03f2877a00a2323"
I20260925 01:33:03.862591 11638 raft_consensus.cc:3037] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:33:03.862568 11504 raft_consensus.cc:3037] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:33:03.865029 11638 raft_consensus.cc:2454] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87909428a991490d80bd73479305f956 in term 1.
I20260925 01:33:03.865540 11706 leader_election.cc:302] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87909428a991490d80bd73479305f956, baa5e7ab94744b32a03f2877a00a2323; no voters:
I20260925 01:33:03.865866 11844 raft_consensus.cc:2781] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:33:03.866264 11504 raft_consensus.cc:2454] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87909428a991490d80bd73479305f956 in term 1.
I20260925 01:33:03.866431 11844 raft_consensus.cc:686] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 1 LEADER]: Becoming Leader. State: Replica: 87909428a991490d80bd73479305f956, State: Running, Role: LEADER
I20260925 01:33:03.867007 11844 consensus_queue.cc:237] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } }
I20260925 01:33:03.871660 11360 catalog_manager.cc:5495] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87909428a991490d80bd73479305f956 (127.27.160.3). New cstate: current_term: 1 leader_uuid: "87909428a991490d80bd73479305f956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } health_report { overall_health: UNKNOWN } } }
I20260925 01:33:03.899463 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.893170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.27.160.3:33397)
0925 01:33:03.893399 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.893404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.893426 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:33:03.893544 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.893548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.893633 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.893857 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:33:03.894269 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.894990 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.896891 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.897435 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.897607 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.898224 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:33:03.898813 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:33:03.899003 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.899060 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.899165 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:33:03.908519 11638 raft_consensus.cc:1261] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 1 FOLLOWER]: Refusing update from remote peer 87909428a991490d80bd73479305f956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:33:03.908519 11504 raft_consensus.cc:1261] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Refusing update from remote peer 87909428a991490d80bd73479305f956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:33:03.909711 11844 consensus_queue.cc:1035] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:33:03.910172 11846 consensus_queue.cc:1035] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:33:03.922976 11849 mvcc.cc:205] Tried to move back new op lower bound from 7333068734078189568 to 7333068733923688448. Current Snapshot: MvccSnapshot[applied={T|T < 7333068734078189568}]
I20260925 01:33:03.938138 11360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:33:03.938444 11360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:33:03.942552 11360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Sending DeleteTablet for 3 replicas of tablet 9baf5fc757e04ba38a458f93c971599e
I20260925 01:33:03.943943 11484 tablet_service.cc:1514] Processing DeleteTablet for tablet 9baf5fc757e04ba38a458f93c971599e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:33:03 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:60364
I20260925 01:33:03.943953 11752 tablet_service.cc:1514] Processing DeleteTablet for tablet 9baf5fc757e04ba38a458f93c971599e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:33:03 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:48672
I20260925 01:33:03.944799 11618 tablet_service.cc:1514] Processing DeleteTablet for tablet 9baf5fc757e04ba38a458f93c971599e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:33:03 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:32812
I20260925 01:33:03.946219 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 11421
I20260925 01:33:03.954984 11860 tablet_replica.cc:330] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: stopping tablet replica
I20260925 01:33:03.955711 11860 raft_consensus.cc:2229] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:33:03.956599 11860 raft_consensus.cc:2258] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:33:03.959827 11860 ts_tablet_manager.cc:1905] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:33:03.960230 11862 tablet_replica.cc:330] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: stopping tablet replica
I20260925 01:33:03.960786 11862 raft_consensus.cc:2229] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:33:03.961436 11862 raft_consensus.cc:2258] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:33:03.962121 11861 tablet_replica.cc:330] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: stopping tablet replica
I20260925 01:33:03.963943 11861 raft_consensus.cc:2229] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:33:03.964696 11862 ts_tablet_manager.cc:1905] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:33:03.965597 11861 raft_consensus.cc:2258] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:33:03.971941 11860 ts_tablet_manager.cc:1918] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:33:03.972163 11862 ts_tablet_manager.cc:1918] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:33:03.972366 11862 log.cc:1196] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/9baf5fc757e04ba38a458f93c971599e
I20260925 01:33:03.972965 11862 ts_tablet_manager.cc:1939] T 9baf5fc757e04ba38a458f93c971599e P baa5e7ab94744b32a03f2877a00a2323: Deleting consensus metadata
I20260925 01:33:03.974951 11861 ts_tablet_manager.cc:1905] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:33:03.975168 11345 catalog_manager.cc:4841] TS baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969): tablet 9baf5fc757e04ba38a458f93c971599e (table test-table [id=07d3832e6bc44cdca1f73f760e793f82]) successfully deleted
I20260925 01:33:03.973261 11860 log.cc:1196] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/9baf5fc757e04ba38a458f93c971599e
Sep 25 01:33:03 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:33:03.977089 11860 ts_tablet_manager.cc:1939] T 9baf5fc757e04ba38a458f93c971599e P 87909428a991490d80bd73479305f956: Deleting consensus metadata
I20260925 01:33:03.979256 11346 catalog_manager.cc:4841] TS 87909428a991490d80bd73479305f956 (127.27.160.3:33397): tablet 9baf5fc757e04ba38a458f93c971599e (table test-table [id=07d3832e6bc44cdca1f73f760e793f82]) successfully deleted
I20260925 01:33:03.983685 11861 ts_tablet_manager.cc:1918] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:33:03.983987 11861 log.cc:1196] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/9baf5fc757e04ba38a458f93c971599e
I20260925 01:33:03.984654 11861 ts_tablet_manager.cc:1939] T 9baf5fc757e04ba38a458f93c971599e P b83f7a4f20d645b08667e14c09c2bb52: Deleting consensus metadata
I20260925 01:33:03.985553 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:03.947715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.27.160.1:40009)
0925 01:33:03.947948 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:03.947958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:03.964865 (+ 16907us) server_negotiation.cc:407] Connection header received
0925 01:33:03.965574 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:03.965584 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:03.965683 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:03.965759 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:03.967869 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.968989 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.971158 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:03.971810 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:03.971935 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:03.978633 (+ 6698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:03.978661 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:03.978679 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:03.978731 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:03.982131 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:03.982926 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:03.982935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:03.982937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:03.983090 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:03.983556 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:03.983561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:03.983565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:03.984014 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:03.984300 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:03.984827 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 01:33:03.985063 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:33:03.986217 11544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:33:04.081450 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 11421
I20260925 01:33:04.081657 11344 catalog_manager.cc:4841] TS b83f7a4f20d645b08667e14c09c2bb52 (127.27.160.1:40009): tablet 9baf5fc757e04ba38a458f93c971599e (table test-table [id=07d3832e6bc44cdca1f73f760e793f82]) successfully deleted
I20260925 01:33:04.109409 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 11555
Sep 25 01:33:04 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:33:04.126281 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:04.110300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.27.160.2:41969)
0925 01:33:04.110500 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:04.110506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:04.110604 (+ 98us) server_negotiation.cc:407] Connection header received
0925 01:33:04.110833 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:04.110839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:04.110915 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:04.111015 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:04.112302 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:04.113049 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:04.114460 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:04.114888 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:04.115024 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:04.119625 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:04.119641 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:04.119658 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:04.119695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:04.123142 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:04.123974 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:04.123986 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:04.123992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:04.124152 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:04.124576 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:04.124582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:04.124585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:04.124992 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:04.125327 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:04.125675 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 01:33:04.125921 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:33:04.127024 11678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:33:04.218338 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 11555
I20260925 01:33:04.236251 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 11689
Sep 25 01:33:04 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:33:04.254997 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:04.237018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.27.160.3:33397)
0925 01:33:04.237230 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:04.237236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:04.237342 (+ 106us) server_negotiation.cc:407] Connection header received
0925 01:33:04.237551 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:04.237561 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:04.237626 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:04.237789 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:04.239062 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:04.239903 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:04.241336 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:04.241804 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:04.241951 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:04.246579 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:04.246593 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:04.246614 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:04.246676 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:04.251308 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:04.252270 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:04.252282 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:04.252286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:04.252536 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:04.252989 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:04.252995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:04.252999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:04.253639 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:04.253974 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:04.254288 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 01:33:04.254558 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 01:33:04.255926 11812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:33:04.378315 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 11689
I20260925 01:33:04.399312 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 11328
I20260925 01:33:04.400681 11390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:33:04.514365 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 11328
I20260925 01:33:24.538193 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37989
--webserver_interface=127.27.160.62
--webserver_port=34457
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:37989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:33:24.884150 11868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:24.884464 11868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:24.884555 11868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:24.891860 11868 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:33:24.891971 11868 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:33:24.892014 11868 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:33:24.892129 11868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:33:24.892261 11868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:24.892393 11868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:24.892455 11868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:33:24.892556 11868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:33:24.902765 11868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:37989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=34457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:24.904151 11868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:24.905583 11868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:24.928637 11877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:24.928741 11874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:24.928717 11875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:24.929445 11868 server_base.cc:1034] running on GCE node
I20260925 01:33:24.930370 11868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:24.931937 11868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:24.933203 11868 hybrid_clock.cc:648] HybridClock initialized: now 1790300004933077 us; error 59 us; skew 500 ppm
I20260925 01:33:24.933622 11868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:24 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300004, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:24.937489 11868 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:33:24.939236 11868 webserver.cc:460] Webserver started at http://127.27.160.62:34457/ using document root <none> and password file <none>
I20260925 01:33:24.940048 11868 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:24.940164 11868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:24.946821 11868 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 01:33:24.950177 11884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:24.951874 11868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:33:24.952083 11868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "10e9c376b8034b5e89947df54c47c767"
format_stamp: "Formatted at 2026-09-25 01:33:01 on dist-test-slave-h6hf"
server_key: "5a3618e954545b886780693dcd217ae7"
server_key_iv: "e324e5b1e4f2097ce83edc642d08eaa8"
server_key_version: "encryptionkey@0"
I20260925 01:33:24.953399 11868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:24.972012 11868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:24.976784 11868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:24.977043 11868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:24.997560 11868 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:37989
I20260925 01:33:24.997584 11935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:37989 every 8 connection(s)
I20260925 01:33:24.999214 11868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:33:25.004573 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 11868
I20260925 01:33:25.005698 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:40009
--local_ip_for_outbound_sockets=127.27.160.1
--tserver_master_addrs=127.27.160.62:37989
--webserver_port=36601
--webserver_interface=127.27.160.1
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:33:25.007777 11936 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:33:25.013340 11936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Bootstrap starting.
I20260925 01:33:25.034512 11936 log.cc:824] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:25.047927 11936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:33:25.048555 11936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Bootstrap complete.
I20260925 01:33:25.058357 11936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:25.059319 11936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e9c376b8034b5e89947df54c47c767, State: Initialized, Role: FOLLOWER
I20260925 01:33:25.059968 11936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:25.060221 11936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:33:25.060341 11936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:33:25.060535 11936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:33:25.064656 11936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:25.065344 11936 leader_election.cc:302] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10e9c376b8034b5e89947df54c47c767; no voters:
I20260925 01:33:25.066066 11936 leader_election.cc:288] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:33:25.066215 11940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:33:25.067348 11940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 2 LEADER]: Becoming Leader. State: Replica: 10e9c376b8034b5e89947df54c47c767, State: Running, Role: LEADER
I20260925 01:33:25.068274 11940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:25.068805 11936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:33:25.071237 11942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10e9c376b8034b5e89947df54c47c767. Latest consensus state: current_term: 2 leader_uuid: "10e9c376b8034b5e89947df54c47c767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } } }
I20260925 01:33:25.071738 11942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: This master's current role is: LEADER
I20260925 01:33:25.071235 11941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "10e9c376b8034b5e89947df54c47c767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } } }
I20260925 01:33:25.072520 11941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: This master's current role is: LEADER
I20260925 01:33:25.073432 11946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:33:25.087412 11946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:33:25.089473 11946 catalog_manager.cc:1259] Loaded cluster ID: 4e6fb7c2483e4ce1bb30e8658a453539
I20260925 01:33:25.089675 11946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:33:25.095252 11946 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:33:25.097479 11946 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Loaded TSK: 0
I20260925 01:33:25.142478 11946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Generated new TSK 1
I20260925 01:33:25.142962 11946 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:33:25.395174 11938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:25.395496 11938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:25.395644 11938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:25.403353 11938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:25.403473 11938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:25.403648 11938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:33:25.414814 11938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:40009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=36601
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:25.416437 11938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:25.418169 11938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:25.436079 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:25.436281 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:25.437461 11938 server_base.cc:1034] running on GCE node
W20260925 01:33:25.441515 11966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:25.442394 11938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:25.443683 11938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:25.444923 11938 hybrid_clock.cc:648] HybridClock initialized: now 1790300005444880 us; error 47 us; skew 500 ppm
I20260925 01:33:25.445323 11938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:25 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300005, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:25.449195 11938 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:33:25.451462 11938 webserver.cc:460] Webserver started at http://127.27.160.1:36601/ using document root <none> and password file <none>
I20260925 01:33:25.452642 11938 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:25.452884 11938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:25.462303 11938 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 01:33:25.466246 11973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:25.468288 11938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:33:25.468523 11938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "b83f7a4f20d645b08667e14c09c2bb52"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "b9dfdf23c382dc0459cf3d33d74abcce"
server_key_iv: "dcac05e2bc97d61c98c4aa715da45592"
server_key_version: "encryptionkey@0"
I20260925 01:33:25.470343 11938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:25.496781 11938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:25.502779 11938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:25.503074 11938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:25.504274 11938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:25.506033 11938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:33:25.506143 11938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:25.506253 11938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:33:25.506325 11938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:25.547848 11938 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:40009
I20260925 01:33:25.547871 12085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:40009 every 8 connection(s)
I20260925 01:33:25.549830 11938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:33:25.556069 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 11938
I20260925 01:33:25.556973 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:41969
--local_ip_for_outbound_sockets=127.27.160.2
--tserver_master_addrs=127.27.160.62:37989
--webserver_port=44885
--webserver_interface=127.27.160.2
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:33:25 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300005, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:33:25.581910 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:25.553269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:58257 (local address 127.27.160.62:37989)
0925 01:33:25.553996 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:25.554017 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:25.554628 (+ 611us) server_negotiation.cc:407] Connection header received
0925 01:33:25.557953 (+ 3325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:25.558063 (+ 110us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:25.558643 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:25.559239 (+ 596us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:25.562665 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:25.564009 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:25.566019 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:25.567283 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:25.567483 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:25.572216 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:25.572227 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:25.572254 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:25.572306 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:25.576013 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:25.577252 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:25.577262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:25.577269 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:25.577471 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:25.578149 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:25.578153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:25.578156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:25.578718 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:25.578966 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:25.579905 (+ 939us) server_negotiation.cc:299] Negotiation successful
0925 01:33:25.580346 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":315,"threads_started":1}
I20260925 01:33:25.581909 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:25.553569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:58257)
0925 01:33:25.554428 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:25.554468 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:25.557029 (+ 2561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:25.559606 (+ 2577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:25.559621 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:25.561688 (+ 2067us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:33:25.562355 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:25.562378 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:25.564279 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:25.564283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:25.565699 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:25.565738 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:25.567867 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:25.567872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:25.568088 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:25.568899 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:33:25.568946 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:33:25.571818 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:33:25.576214 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:25.576235 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:25.576254 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:25.576914 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:25.577715 (+ 801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:25.577719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:25.577722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:25.577986 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:25.578892 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:33:25.578902 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:33:25.579325 (+ 423us) client_negotiation.cc:769] Sending connection context
0925 01:33:25.579649 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 01:33:25.580344 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":347,"threads_started":1}
I20260925 01:33:25.585690 12086 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37989
I20260925 01:33:25.586158 12086 heartbeater.cc:464] Registering TS with master...
I20260925 01:33:25.586993 12086 heartbeater.cc:511] Master 127.27.160.62:37989 requested a full tablet report, sending...
I20260925 01:33:25.589679 11901 ts_manager.cc:194] Registered new tserver with Master: b83f7a4f20d645b08667e14c09c2bb52 (127.27.160.1:40009)
I20260925 01:33:25.591902 11901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:58257
W20260925 01:33:25.916903 12090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:25.917310 12090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:25.917418 12090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:25.926071 12090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:25.926183 12090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:25.926299 12090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:33:25.936573 12090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:41969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=44885
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:25.937991 12090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:25.939328 12090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:25.953086 12095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:25.953189 12098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:25.953037 12096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:25.954406 12090 server_base.cc:1034] running on GCE node
I20260925 01:33:25.955091 12090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:25.956049 12090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:25.957307 12090 hybrid_clock.cc:648] HybridClock initialized: now 1790300005957262 us; error 45 us; skew 500 ppm
I20260925 01:33:25.957652 12090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:25 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300005, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:25.961058 12090 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:33:25.962630 12090 webserver.cc:460] Webserver started at http://127.27.160.2:44885/ using document root <none> and password file <none>
I20260925 01:33:25.963441 12090 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:25.963606 12090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:25.969458 12090 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:33:25.972648 12105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:25.974758 12090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:33:25.974963 12090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "baa5e7ab94744b32a03f2877a00a2323"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "0f3f83a9982cad11fe7406641c83271b"
server_key_iv: "9ac7014cf12c3b2e1b58e7b79a35c333"
server_key_version: "encryptionkey@0"
I20260925 01:33:25.976552 12090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:26.003571 12090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:26.008488 12090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:26.008800 12090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:26.009938 12090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:26.011440 12090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:33:26.011591 12090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:26.011722 12090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:33:26.011811 12090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:26.048740 12090 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:41969
I20260925 01:33:26.048837 12217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:41969 every 8 connection(s)
I20260925 01:33:26.050410 12090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:33:26.057060 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 12090
I20260925 01:33:26.057914 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:33397
--local_ip_for_outbound_sockets=127.27.160.3
--tserver_master_addrs=127.27.160.62:37989
--webserver_port=45663
--webserver_interface=127.27.160.3
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:33:26 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300005, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:33:26.076517 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.053398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:53225 (local address 127.27.160.62:37989)
0925 01:33:26.053617 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.053623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.054883 (+ 1260us) server_negotiation.cc:407] Connection header received
0925 01:33:26.056926 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.056933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.057061 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.057232 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:26.059727 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.060909 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.062951 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.063717 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.063874 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.068016 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:26.068025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:26.068033 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:26.068083 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:26.072100 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.073098 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.073107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.073113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.073371 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.073979 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.073988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.073994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.074445 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:26.074597 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.075711 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.075985 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:33:26.077742 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.053929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:53225)
0925 01:33:26.054692 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.054741 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:26.056592 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:26.057626 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:26.057641 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:26.058612 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:33:26.059490 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.059517 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.061188 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.061195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.062760 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.062779 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.063980 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.063986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.064217 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.065022 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:33:26.065047 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:33:26.067731 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:33:26.072319 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:26.072329 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:26.072347 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:26.072922 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:26.073534 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:26.073539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:26.073541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:26.073816 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:26.074700 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:33:26.074708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:33:26.075116 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 01:33:26.075482 (+ 366us) client_negotiation.cc:240] Negotiation successful
0925 01:33:26.076114 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":303,"threads_started":1}
I20260925 01:33:26.079815 12218 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37989
I20260925 01:33:26.080179 12218 heartbeater.cc:464] Registering TS with master...
I20260925 01:33:26.081176 12218 heartbeater.cc:511] Master 127.27.160.62:37989 requested a full tablet report, sending...
I20260925 01:33:26.083048 11901 ts_manager.cc:194] Registered new tserver with Master: baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969)
I20260925 01:33:26.084441 11901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:53225
W20260925 01:33:26.402374 12221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:26.402671 12221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:26.402776 12221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:26.410332 12221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:26.410468 12221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:26.410590 12221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:33:26.420693 12221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:33397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=45663
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:26.422060 12221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:26.423341 12221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:26.437551 12226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:26.437486 12229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:26.437528 12227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:26.438869 12221 server_base.cc:1034] running on GCE node
I20260925 01:33:26.439519 12221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:26.440572 12221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:26.441792 12221 hybrid_clock.cc:648] HybridClock initialized: now 1790300006441713 us; error 62 us; skew 500 ppm
I20260925 01:33:26.442149 12221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:26 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:26.445418 12221 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:33:26.446995 12221 webserver.cc:460] Webserver started at http://127.27.160.3:45663/ using document root <none> and password file <none>
I20260925 01:33:26.447677 12221 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:26.447798 12221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:26.453543 12221 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 01:33:26.456490 12236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:26.458297 12221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:33:26.458495 12221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "87909428a991490d80bd73479305f956"
format_stamp: "Formatted at 2026-09-25 01:33:03 on dist-test-slave-h6hf"
server_key: "ba993ea6dee43634bdd5dcc7b84135e5"
server_key_iv: "38d613830114ecd494e68508ed76ebca"
server_key_version: "encryptionkey@0"
I20260925 01:33:26.459600 12221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:26.482311 12221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:26.487011 12221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:26.487324 12221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:26.488382 12221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:26.489840 12221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:33:26.489970 12221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:26.490108 12221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:33:26.490195 12221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:26.525032 12221 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:33397
I20260925 01:33:26.525091 12348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:33397 every 8 connection(s)
I20260925 01:33:26.526542 12221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:33:26.527247 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 12221
Sep 25 01:33:26 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300006, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:33:26.543927 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.528472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.27.160.62:37989)
0925 01:33:26.528641 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.528646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.529215 (+ 569us) server_negotiation.cc:407] Connection header received
0925 01:33:26.529447 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.529452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.529546 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.529673 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:26.531509 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.532254 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.534037 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.534481 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.534625 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.537704 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:26.537709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:26.537713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:26.537752 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:26.540654 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.541525 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.541530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.541532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.541668 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.542208 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.542213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.542215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.542547 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:26.542939 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.543158 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.543433 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 01:33:26.549948 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.529625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:38597 (local address 127.27.160.62:37989)
0925 01:33:26.530238 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.530244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.531391 (+ 1147us) server_negotiation.cc:407] Connection header received
0925 01:33:26.533277 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.533282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.533377 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.533480 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:26.535119 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.536249 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.538249 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.538773 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.538932 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.542843 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:26.542851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:26.542855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:26.542896 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:26.546250 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.547248 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.547254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.547262 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.547380 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.548006 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.548010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.548013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.548407 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:26.548563 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.549321 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.549563 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":349,"threads_started":1}
I20260925 01:33:26.550952 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.530365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:38597)
0925 01:33:26.531216 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.531260 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:26.532989 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:26.533772 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:26.533781 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:26.534348 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:33:26.534935 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.534953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.536477 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.536482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.538068 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.538082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.539057 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.539069 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.539356 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.539958 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:33:26.539988 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:33:26.542554 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:33:26.546417 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:26.546428 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:26.546448 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:26.547090 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:26.547504 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:26.547510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:26.547514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:26.547849 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:26.548541 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:33:26.548548 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:33:26.548905 (+ 357us) client_negotiation.cc:769] Sending connection context
0925 01:33:26.549218 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 01:33:26.549679 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":373,"threads_started":1}
I20260925 01:33:26.552619 12349 heartbeater.cc:347] Connected to a master server at 127.27.160.62:37989
I20260925 01:33:26.553025 12349 heartbeater.cc:464] Registering TS with master...
I20260925 01:33:26.553871 12349 heartbeater.cc:511] Master 127.27.160.62:37989 requested a full tablet report, sending...
I20260925 01:33:26.555401 11901 ts_manager.cc:194] Registered new tserver with Master: 87909428a991490d80bd73479305f956 (127.27.160.3:33397)
I20260925 01:33:26.556298 11901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:38597
I20260925 01:33:26.559336 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:33:26.566972 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.560556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.27.160.62:37989)
0925 01:33:26.560793 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.560800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.561023 (+ 223us) server_negotiation.cc:407] Connection header received
0925 01:33:26.561326 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.561330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.561474 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.561611 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:33:26.562146 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.563014 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.564689 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.565348 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.565464 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.566042 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:33:26.566310 (+ 268us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 01:33:26.566640 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57604: token expired
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 01:33:26.567155 12352 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57604: token expired
W20260925 01:33:26.567708 11822 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.160.62:37989)
I20260925 01:33:26.581586 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.568228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.27.160.62:37989)
0925 01:33:26.568444 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.568449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.568467 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:33:26.568611 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.568616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.568687 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.568792 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:26.570286 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.571046 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.572422 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.572854 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.572991 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.575761 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:26.575766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:26.575769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:26.575806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:26.578758 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.579460 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.579467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.579470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.579602 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.579991 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.579997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.580000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.580339 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:26.580645 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.580963 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.581230 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:33:26.585399 11901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:33:26.592967 11901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:33:26.595780 12086 heartbeater.cc:503] Master 127.27.160.62:37989 was elected leader, sending a full tablet report...
I20260925 01:33:26.620777 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.606888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.27.160.2:41969)
0925 01:33:26.609298 (+ 2410us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.609306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.609335 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:33:26.609436 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.609445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.609722 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.610882 (+ 1160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:26.611164 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.613451 (+ 2287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.617712 (+ 4261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.619236 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.619439 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.619696 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.620186 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.620380 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2254,"thread_start_us":2141,"threads_started":1}
I20260925 01:33:26.620946 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.606555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41969 (local address 127.0.0.1:39290)
0925 01:33:26.607514 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.607571 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:26.607764 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:26.609988 (+ 2224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:26.609997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:26.610033 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:26.610581 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.610599 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.613790 (+ 3191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.613800 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.617382 (+ 3582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.617401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.619542 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.619550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.619891 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.619963 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 01:33:26.620105 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 01:33:26.620451 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":28,"thread_start_us":484,"threads_started":1}
I20260925 01:33:26.625241 12153 tablet_service.cc:1467] Processing CreateTablet for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8 (DEFAULT_TABLE table=test-table [id=56210e33a45249fcaf490c2123719357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:33:26.626860 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.606865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56066)
0925 01:33:26.617332 (+ 10467us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.617359 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:26.618029 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:26.618855 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:26.618858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:26.618882 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:26.619189 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.619198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.621621 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.621633 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.623657 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.623680 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.625918 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.625924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.626225 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.626266 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 01:33:26.626362 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 01:33:26.626499 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10290,"thread_start_us":10054,"threads_started":1}
I20260925 01:33:26.626971 12153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:33:26.627256 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.607503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.27.160.1:40009)
0925 01:33:26.608849 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.608862 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.617517 (+ 8655us) server_negotiation.cc:407] Connection header received
0925 01:33:26.618257 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.618267 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.618559 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.618822 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:26.619368 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.621436 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.623845 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.625371 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.625506 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.625748 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.626665 (+ 917us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.626868 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":1023,"threads_started":1}
I20260925 01:33:26.628021 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.609411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:33397 (local address 127.0.0.1:38276)
0925 01:33:26.618019 (+ 8608us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.618039 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:26.618159 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:26.619794 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:26.619800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:26.619832 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:26.621574 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.621591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.623165 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.623173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.625097 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.625113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.627358 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.627368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.627549 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.627591 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 01:33:26.627694 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 01:33:26.627794 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8452,"thread_start_us":8298,"threads_started":1}
I20260925 01:33:26.628772 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.610133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.27.160.3:33397)
0925 01:33:26.617761 (+ 7628us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.617774 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.619110 (+ 1336us) server_negotiation.cc:407] Connection header received
0925 01:33:26.619229 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.619237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.619495 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.619708 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:26.621802 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.623018 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.625330 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.626786 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.626937 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.627182 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.628371 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.628504 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7426,"thread_start_us":7243,"threads_started":1}
I20260925 01:33:26.631655 12021 tablet_service.cc:1467] Processing CreateTablet for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8 (DEFAULT_TABLE table=test-table [id=56210e33a45249fcaf490c2123719357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:33:26.632715 12284 tablet_service.cc:1467] Processing CreateTablet for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8 (DEFAULT_TABLE table=test-table [id=56210e33a45249fcaf490c2123719357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:33:26.633525 12021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:33:26.634265 12284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:33:26.644758 12365 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: Bootstrap starting.
I20260925 01:33:26.647146 12366 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: Bootstrap starting.
I20260925 01:33:26.649206 12365 tablet_bootstrap.cc:654] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: Neither blocks nor log segments found. Creating new log.
I20260925 01:33:26.650599 12367 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Bootstrap starting.
I20260925 01:33:26.650854 12365 log.cc:824] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:26.651317 12366 tablet_bootstrap.cc:654] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: Neither blocks nor log segments found. Creating new log.
I20260925 01:33:26.652766 12366 log.cc:824] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:26.654264 12367 tablet_bootstrap.cc:654] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Neither blocks nor log segments found. Creating new log.
I20260925 01:33:26.654788 12365 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: No bootstrap required, opened a new log
I20260925 01:33:26.655118 12365 ts_tablet_manager.cc:1397] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 01:33:26.655928 12366 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: No bootstrap required, opened a new log
I20260925 01:33:26.655990 12367 log.cc:824] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:26.656286 12366 ts_tablet_manager.cc:1397] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260925 01:33:26.664666 12367 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: No bootstrap required, opened a new log
I20260925 01:33:26.665004 12367 ts_tablet_manager.cc:1397] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260925 01:33:26.666152 12366 raft_consensus.cc:348] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.666720 12366 raft_consensus.cc:374] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:33:26.666859 12366 raft_consensus.cc:729] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87909428a991490d80bd73479305f956, State: Initialized, Role: FOLLOWER
I20260925 01:33:26.667309 12365 raft_consensus.cc:348] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.667788 12365 raft_consensus.cc:374] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:33:26.667899 12365 raft_consensus.cc:729] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baa5e7ab94744b32a03f2877a00a2323, State: Initialized, Role: FOLLOWER
I20260925 01:33:26.667783 12366 consensus_queue.cc:260] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.668646 12365 consensus_queue.cc:260] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.669369 12349 heartbeater.cc:503] Master 127.27.160.62:37989 was elected leader, sending a full tablet report...
I20260925 01:33:26.670132 12366 ts_tablet_manager.cc:1428] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260925 01:33:26.670604 12365 ts_tablet_manager.cc:1428] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260925 01:33:26.670635 12218 heartbeater.cc:503] Master 127.27.160.62:37989 was elected leader, sending a full tablet report...
I20260925 01:33:26.678256 12367 raft_consensus.cc:348] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.678669 12367 raft_consensus.cc:374] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:33:26.678750 12367 raft_consensus.cc:729] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83f7a4f20d645b08667e14c09c2bb52, State: Initialized, Role: FOLLOWER
I20260925 01:33:26.679270 12367 consensus_queue.cc:260] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.681162 12367 ts_tablet_manager.cc:1428] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260925 01:33:26.687997 12371 raft_consensus.cc:482] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:33:26.688352 12371 raft_consensus.cc:504] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.690306 12371 leader_election.cc:288] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969), b83f7a4f20d645b08667e14c09c2bb52 (127.27.160.1:40009)
I20260925 01:33:26.696135 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.691027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:34207 (local address 127.27.160.2:41969)
0925 01:33:26.691251 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.691257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.691278 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:33:26.691355 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.691359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.691418 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.691532 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:26.691993 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.692714 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.694402 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.695223 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.695390 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.695464 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.695753 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.695888 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:33:26.696136 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.690464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41969 (local address 127.27.160.3:34207)
0925 01:33:26.690667 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.690688 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:26.690813 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:26.691568 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:26.691573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:26.691609 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:26.691834 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.691843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.692851 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.692858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.694233 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.694245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.695382 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.695388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.695573 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.695598 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:33:26.695703 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:33:26.695836 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260925 01:33:26.697291 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.691071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:50963 (local address 127.27.160.1:40009)
0925 01:33:26.691274 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.691279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.691970 (+ 691us) server_negotiation.cc:407] Connection header received
0925 01:33:26.692171 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.692175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.692233 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.692330 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:33:26.692840 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.693624 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.695339 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.696101 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.696288 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.696374 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.696867 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.697013 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:33:26.697352 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.690659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.27.160.3:50963)
0925 01:33:26.691863 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.691879 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:26.692030 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:26.692344 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:26.692348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:26.692371 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:33:26.692688 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.692697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.693761 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.693767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.695179 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:26.695191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.696583 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.696587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:26.696705 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.696729 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:33:26.696841 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 01:33:26.696973 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":786,"threads_started":1}
I20260925 01:33:26.697242 12173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bd3bbadce4b1a8e0afd12c5dd87a8" candidate_uuid: "87909428a991490d80bd73479305f956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7ab94744b32a03f2877a00a2323" is_pre_election: true
I20260925 01:33:26.697781 12173 raft_consensus.cc:2454] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87909428a991490d80bd73479305f956 in term 0.
I20260925 01:33:26.698452 12238 leader_election.cc:302] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87909428a991490d80bd73479305f956, baa5e7ab94744b32a03f2877a00a2323; no voters:
I20260925 01:33:26.698423 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bd3bbadce4b1a8e0afd12c5dd87a8" candidate_uuid: "87909428a991490d80bd73479305f956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83f7a4f20d645b08667e14c09c2bb52" is_pre_election: true
I20260925 01:33:26.698874 12371 raft_consensus.cc:2781] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:33:26.699041 12371 raft_consensus.cc:482] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:33:26.698982 12041 raft_consensus.cc:2454] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87909428a991490d80bd73479305f956 in term 0.
I20260925 01:33:26.699167 12371 raft_consensus.cc:3037] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:33:26.701666 12371 raft_consensus.cc:504] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.702294 12371 leader_election.cc:288] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [CANDIDATE]: Term 1 election: Requested vote from peers baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969), b83f7a4f20d645b08667e14c09c2bb52 (127.27.160.1:40009)
I20260925 01:33:26.702752 12173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bd3bbadce4b1a8e0afd12c5dd87a8" candidate_uuid: "87909428a991490d80bd73479305f956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baa5e7ab94744b32a03f2877a00a2323"
I20260925 01:33:26.702804 12041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "417bd3bbadce4b1a8e0afd12c5dd87a8" candidate_uuid: "87909428a991490d80bd73479305f956" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b83f7a4f20d645b08667e14c09c2bb52"
I20260925 01:33:26.702993 12041 raft_consensus.cc:3037] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:33:26.702993 12173 raft_consensus.cc:3037] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:33:26.706398 12173 raft_consensus.cc:2454] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87909428a991490d80bd73479305f956 in term 1.
I20260925 01:33:26.706398 12041 raft_consensus.cc:2454] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87909428a991490d80bd73479305f956 in term 1.
I20260925 01:33:26.706967 12237 leader_election.cc:302] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87909428a991490d80bd73479305f956, b83f7a4f20d645b08667e14c09c2bb52; no voters:
I20260925 01:33:26.707373 12371 raft_consensus.cc:2781] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:33:26.707824 12371 raft_consensus.cc:686] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 1 LEADER]: Becoming Leader. State: Replica: 87909428a991490d80bd73479305f956, State: Running, Role: LEADER
I20260925 01:33:26.708405 12371 consensus_queue.cc:237] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:26.713588 11899 catalog_manager.cc:5495] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87909428a991490d80bd73479305f956 (127.27.160.3). New cstate: current_term: 1 leader_uuid: "87909428a991490d80bd73479305f956" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } health_report { overall_health: HEALTHY } } }
I20260925 01:33:26.748227 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.742485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.27.160.3:33397)
0925 01:33:26.742700 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.742705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.742757 (+ 52us) server_negotiation.cc:407] Connection header received
0925 01:33:26.742852 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.742856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.742940 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.743080 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:33:26.743427 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.744210 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.745711 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.746143 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.746255 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.746847 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:33:26.747509 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:33:26.747648 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.747811 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.747893 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:33:26.757498 12041 raft_consensus.cc:1261] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Refusing update from remote peer 87909428a991490d80bd73479305f956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:33:26.757628 12173 raft_consensus.cc:1261] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 1 FOLLOWER]: Refusing update from remote peer 87909428a991490d80bd73479305f956: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:33:26.758710 12375 consensus_queue.cc:1035] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:33:26.759217 12371 consensus_queue.cc:1035] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:33:26.773258 12377 mvcc.cc:205] Tried to move back new op lower bound from 7333068827667922944 to 7333068827482116096. Current Snapshot: MvccSnapshot[applied={T|T < 7333068827667922944}]
I20260925 01:33:26.773764 12379 mvcc.cc:205] Tried to move back new op lower bound from 7333068827667922944 to 7333068827482116096. Current Snapshot: MvccSnapshot[applied={T|T < 7333068827667922944}]
I20260925 01:33:26.788139 11900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:33:26.788450 11900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:33:26.792356 11900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Sending DeleteTablet for 3 replicas of tablet 417bd3bbadce4b1a8e0afd12c5dd87a8
I20260925 01:33:26.793713 12284 tablet_service.cc:1514] Processing DeleteTablet for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:33:26 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:38276
I20260925 01:33:26.793708 12021 tablet_service.cc:1514] Processing DeleteTablet for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:33:26 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:56066
I20260925 01:33:26.793888 12153 tablet_service.cc:1514] Processing DeleteTablet for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:33:26 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:39290
I20260925 01:33:26.795975 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 11938
I20260925 01:33:26.806041 12388 tablet_replica.cc:330] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: stopping tablet replica
I20260925 01:33:26.806588 12388 raft_consensus.cc:2229] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:33:26.807451 12388 raft_consensus.cc:2258] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:33:26.812409 12388 ts_tablet_manager.cc:1905] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:33:26.813771 12389 tablet_replica.cc:330] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: stopping tablet replica
I20260925 01:33:26.814450 12389 raft_consensus.cc:2229] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:33:26.815096 12389 raft_consensus.cc:2258] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:33:26.816622 12387 tablet_replica.cc:330] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: stopping tablet replica
I20260925 01:33:26.817241 12387 raft_consensus.cc:2229] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:33:26.817387 12389 ts_tablet_manager.cc:1905] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:33:26.817869 12387 raft_consensus.cc:2258] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:33:26.819736 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.797538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.27.160.1:40009)
0925 01:33:26.797755 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.797764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.797788 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:33:26.797862 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.797869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.797971 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.798125 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:26.800430 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.802064 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.804801 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.805748 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.805895 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.810681 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:26.810707 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:26.810756 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:26.810846 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:26.815046 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.816166 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.816185 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.816191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.816406 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.816902 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.816907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.816912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.817505 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:26.818984 (+ 1479us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.819128 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.819374 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:33:26.820456 12081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:33:26.823612 12388 ts_tablet_manager.cc:1918] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:33:26.823874 12388 log.cc:1196] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/417bd3bbadce4b1a8e0afd12c5dd87a8
I20260925 01:33:26.824604 12388 ts_tablet_manager.cc:1939] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P 87909428a991490d80bd73479305f956: Deleting consensus metadata
I20260925 01:33:26.826786 11887 catalog_manager.cc:4841] TS 87909428a991490d80bd73479305f956 (127.27.160.3:33397): tablet 417bd3bbadce4b1a8e0afd12c5dd87a8 (table test-table [id=56210e33a45249fcaf490c2123719357]) successfully deleted
I20260925 01:33:26.827652 12389 ts_tablet_manager.cc:1918] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:33:26.827870 12389 log.cc:1196] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/417bd3bbadce4b1a8e0afd12c5dd87a8
I20260925 01:33:26.828673 12389 ts_tablet_manager.cc:1939] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P baa5e7ab94744b32a03f2877a00a2323: Deleting consensus metadata
I20260925 01:33:26.831164 11886 catalog_manager.cc:4841] TS baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969): tablet 417bd3bbadce4b1a8e0afd12c5dd87a8 (table test-table [id=56210e33a45249fcaf490c2123719357]) successfully deleted
I20260925 01:33:26.920522 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 11938
I20260925 01:33:26.939062 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 12090
W20260925 01:33:26.939126 11885 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.160.1:40009 (error 108)
I20260925 01:33:26.941388 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.940792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56082)
0925 01:33:26.941047 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.941195 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260925 01:33:26.941710 11885 catalog_manager.cc:4584] TS b83f7a4f20d645b08667e14c09c2bb52 (127.27.160.1:40009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 417bd3bbadce4b1a8e0afd12c5dd87a8: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
I20260925 01:33:26.954067 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.940425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.27.160.2:41969)
0925 01:33:26.940618 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:26.940625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:26.940646 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:33:26.940714 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:26.940719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:26.940800 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:26.940915 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:26.942544 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.943293 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.944621 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:26.945032 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:26.945225 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:26.947954 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:26.947971 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:26.947988 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:26.948025 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:26.951071 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.951859 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.951867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.951869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.951999 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:26.952380 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:26.952385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:26.952388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:26.952775 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:26.953083 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:26.953471 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 01:33:26.953700 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:33:26.954905 12213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:33:26.995585 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.994987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56096)
0925 01:33:26.995251 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.995388 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 01:33:26.996798 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:26.996372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56102)
0925 01:33:26.996544 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:26.996633 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 01:33:27.048003 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 12090
I20260925 01:33:27.057615 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:27.056979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56106)
0925 01:33:27.057281 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:27.057410 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 01:33:27.059123 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:27.058610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56118)
0925 01:33:27.058814 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:27.058916 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 01:33:27.067041 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 12221
I20260925 01:33:27.085806 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:27.067915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.27.160.3:33397)
0925 01:33:27.068095 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:27.068102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:27.068194 (+ 92us) server_negotiation.cc:407] Connection header received
0925 01:33:27.068404 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:27.068408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:27.068513 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:27.068606 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:27.070695 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:27.071537 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:27.073659 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:27.074114 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:27.074221 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:27.077979 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:27.077993 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:27.078015 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:27.078143 (+ 128us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:27.082142 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:27.083280 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:27.083288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:27.083290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:27.083492 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:27.083976 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:27.083980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:27.083982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:27.084426 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:27.084703 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:27.085059 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 01:33:27.085404 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:33:27.086650 12344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:33:27.140156 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:27.139564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56134)
0925 01:33:27.139858 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:27.139977 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260925 01:33:27.141626 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:27.141200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56146)
0925 01:33:27.141412 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:27.141497 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 01:33:27.220510 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 12221
I20260925 01:33:27.240875 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 11868
I20260925 01:33:27.242187 11931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:33:27.360360 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:27.359610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40009 (local address 127.0.0.1:56150)
0925 01:33:27.359926 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:27.360054 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260925 01:33:27.360386 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 11868
I20260925 01:33:47.380420 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37989
--webserver_interface=127.27.160.62
--webserver_port=34457
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:37989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:33:47.726089 12395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:47.726444 12395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:47.726619 12395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:47.733994 12395 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:33:47.734148 12395 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:33:47.734223 12395 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:33:47.734301 12395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:33:47.734409 12395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:47.734529 12395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:47.734624 12395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:33:47.734709 12395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:33:47.745044 12395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:37989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:37989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=34457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:47.746443 12395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:47.747819 12395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:47.760484 12400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:47.760576 12401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:47.760524 12403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:47.761029 12395 server_base.cc:1034] running on GCE node
I20260925 01:33:47.762027 12395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:47.763835 12395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:47.765087 12395 hybrid_clock.cc:648] HybridClock initialized: now 1790300027765023 us; error 38 us; skew 500 ppm
I20260925 01:33:47.765480 12395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:47 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300027, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:47.774762 12395 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:33:47.776415 12395 webserver.cc:460] Webserver started at http://127.27.160.62:34457/ using document root <none> and password file <none>
I20260925 01:33:47.777263 12395 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:47.777403 12395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:47.783176 12395 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:33:47.786326 12410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:47.787811 12395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:33:47.788018 12395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "10e9c376b8034b5e89947df54c47c767"
format_stamp: "Formatted at 2026-09-25 01:33:01 on dist-test-slave-h6hf"
server_key: "5a3618e954545b886780693dcd217ae7"
server_key_iv: "e324e5b1e4f2097ce83edc642d08eaa8"
server_key_version: "encryptionkey@0"
I20260925 01:33:47.789006 12395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:47.807083 12395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:47.811585 12395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:47.811893 12395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:47.831614 12395 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:37989
I20260925 01:33:47.831658 12461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:37989 every 8 connection(s)
I20260925 01:33:47.833254 12395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:33:47.837064 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 12395
I20260925 01:33:47.838049 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:40009
--local_ip_for_outbound_sockets=127.27.160.1
--tserver_master_addrs=127.27.160.62:37989
--webserver_port=36601
--webserver_interface=127.27.160.1
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:33:47.841179 12462 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:33:47.844969 12462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Bootstrap starting.
I20260925 01:33:47.873387 12462 log.cc:824] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:47.890967 12462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:33:47.891621 12462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Bootstrap complete.
I20260925 01:33:47.902170 12462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:47.903215 12462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e9c376b8034b5e89947df54c47c767, State: Initialized, Role: FOLLOWER
I20260925 01:33:47.903872 12462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:47.904151 12462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:33:47.904310 12462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:33:47.904513 12462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 2 FOLLOWER]: Advancing to term 3
I20260925 01:33:47.908762 12462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:47.909354 12462 leader_election.cc:302] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10e9c376b8034b5e89947df54c47c767; no voters:
I20260925 01:33:47.910100 12462 leader_election.cc:288] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 01:33:47.910274 12466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 01:33:47.911386 12466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [term 3 LEADER]: Becoming Leader. State: Replica: 10e9c376b8034b5e89947df54c47c767, State: Running, Role: LEADER
I20260925 01:33:47.912016 12466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } }
I20260925 01:33:47.912516 12462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:33:47.914662 12468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10e9c376b8034b5e89947df54c47c767. Latest consensus state: current_term: 3 leader_uuid: "10e9c376b8034b5e89947df54c47c767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } } }
I20260925 01:33:47.915099 12468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: This master's current role is: LEADER
I20260925 01:33:47.915756 12467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "10e9c376b8034b5e89947df54c47c767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10e9c376b8034b5e89947df54c47c767" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 37989 } } }
I20260925 01:33:47.916040 12467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767 [sys.catalog]: This master's current role is: LEADER
I20260925 01:33:47.919940 12472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:33:47.931985 12472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:33:47.934196 12472 catalog_manager.cc:1259] Loaded cluster ID: 4e6fb7c2483e4ce1bb30e8658a453539
I20260925 01:33:47.934342 12472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:33:47.940353 12472 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:33:47.943146 12472 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Loaded TSK: 0
I20260925 01:33:47.943377 12472 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10e9c376b8034b5e89947df54c47c767: Loaded TSK: 1
F20260925 01:33:47.945019 12472 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe9b82ea9ad google::LogMessage::Fail() at ??:0
@ 0x7fe9b82ed88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe9b82ea4de google::LogMessage::Flush() at ??:0
@ 0x7fe9b82eba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe9bc04e7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe9bc03d73e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260925 01:33:47.966298 12484 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fe9cdd191c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe9cdd18783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe9cdda42a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe9cdda40c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe9cddee5e9 std::function<>::operator()() at ??:0
@ 0x7fe9cdd1ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe9cdd1ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe9cdda2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe9cdda2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe9baf36345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe9baf3991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe9baf39799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe9baf14eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe9be1ee6db start_thread at ??:0
@ 0x7fe9b718461f clone at ??:0
W20260925 01:33:48.244746 12464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:48.245036 12464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:48.245217 12464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:48.252564 12464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:48.252682 12464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:48.252897 12464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:33:48.263101 12464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:40009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=36601
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:48.264511 12464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:48.265956 12464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:48.280267 12512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:48.280339 12510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:48.280649 12509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:48.281442 12464 server_base.cc:1034] running on GCE node
I20260925 01:33:48.282081 12464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:48.283066 12464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:48.284317 12464 hybrid_clock.cc:648] HybridClock initialized: now 1790300028284241 us; error 57 us; skew 500 ppm
I20260925 01:33:48.284638 12464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:48 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:48.288115 12464 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:33:48.289775 12464 webserver.cc:460] Webserver started at http://127.27.160.1:36601/ using document root <none> and password file <none>
I20260925 01:33:48.290679 12464 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:48.290848 12464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:48.297287 12464 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 01:33:48.300631 12519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:48.302233 12464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:33:48.302438 12464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "b83f7a4f20d645b08667e14c09c2bb52"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "b9dfdf23c382dc0459cf3d33d74abcce"
server_key_iv: "dcac05e2bc97d61c98c4aa715da45592"
server_key_version: "encryptionkey@0"
I20260925 01:33:48.303529 12464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:48.325448 12464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:48.329911 12464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:48.330210 12464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:48.331342 12464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:48.333427 12526 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 01:33:48.339195 12464 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 01:33:48.339370 12464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 01:33:48.339548 12464 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 01:33:48.343190 12464 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 01:33:48.343327 12464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260925 01:33:48.343677 12526 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Bootstrap starting.
I20260925 01:33:48.352093 12526 log.cc:824] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Log is configured to *not* fsync() on all Append() calls
I20260925 01:33:48.367916 12526 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:33:48.368861 12526 tablet_bootstrap.cc:492] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Bootstrap complete.
I20260925 01:33:48.369483 12526 ts_tablet_manager.cc:1397] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.005s
I20260925 01:33:48.381009 12526 raft_consensus.cc:348] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:48.382915 12526 raft_consensus.cc:729] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83f7a4f20d645b08667e14c09c2bb52, State: Initialized, Role: FOLLOWER
I20260925 01:33:48.384205 12526 consensus_queue.cc:260] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:48.386570 12526 ts_tablet_manager.cc:1428] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260925 01:33:48.392189 12464 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:40009
I20260925 01:33:48.392304 12634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:40009 every 8 connection(s)
I20260925 01:33:48.393685 12464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:33:48.398850 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:48.396846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:43151)
0925 01:33:48.397743 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:48.397976 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":413,"threads_started":1}
W20260925 01:33:48.399555 12635 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:37989 (0 consecutive failures): Network error: Failed to ping master at 127.27.160.62:37989: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
I20260925 01:33:48.400446 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:48.400098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:39947)
0925 01:33:48.400246 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:48.400313 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 01:33:48.401386 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 12464
I20260925 01:33:48.401487 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:48.401104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:36475)
0925 01:33:48.401277 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:48.401349 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260925 01:33:48.401788 12635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:33:48.402256 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:41969
--local_ip_for_outbound_sockets=127.27.160.2
--tserver_master_addrs=127.27.160.62:37989
--webserver_port=44885
--webserver_interface=127.27.160.2
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:33:48.745836 12638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:48.746120 12638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:48.746294 12638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:48.753646 12638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:48.753767 12638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:48.753942 12638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:33:48.764070 12638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:41969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=44885
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:48.765450 12638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:48.766794 12638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:48.781474 12644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:48.781579 12646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:48.781554 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:48.782066 12638 server_base.cc:1034] running on GCE node
I20260925 01:33:48.782972 12638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:48.784009 12638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:48.785261 12638 hybrid_clock.cc:648] HybridClock initialized: now 1790300028785208 us; error 50 us; skew 500 ppm
I20260925 01:33:48.785573 12638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:48 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:48.788882 12638 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:33:48.790454 12638 webserver.cc:460] Webserver started at http://127.27.160.2:44885/ using document root <none> and password file <none>
I20260925 01:33:48.791138 12638 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:48.791265 12638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:48.797710 12638 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:33:48.800902 12653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:48.802724 12638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:33:48.802948 12638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "baa5e7ab94744b32a03f2877a00a2323"
format_stamp: "Formatted at 2026-09-25 01:33:02 on dist-test-slave-h6hf"
server_key: "0f3f83a9982cad11fe7406641c83271b"
server_key_iv: "9ac7014cf12c3b2e1b58e7b79a35c333"
server_key_version: "encryptionkey@0"
I20260925 01:33:48.804071 12638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:48.826887 12638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:48.832062 12638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:48.832404 12638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:48.833595 12638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:48.835129 12638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:33:48.835237 12638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:48.835422 12638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:33:48.835526 12638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:48.873441 12638 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:41969
I20260925 01:33:48.873485 12765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:41969 every 8 connection(s)
I20260925 01:33:48.875099 12638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:33:48.878649 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 12638
I20260925 01:33:48.879356 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:33397
--local_ip_for_outbound_sockets=127.27.160.3
--tserver_master_addrs=127.27.160.62:37989
--webserver_port=45663
--webserver_interface=127.27.160.3
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:33:48.880918 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:48.878629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:60835)
0925 01:33:48.879639 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:48.879970 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":380,"threads_started":1}
W20260925 01:33:48.881770 12766 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:37989 (0 consecutive failures): Network error: Failed to ping master at 127.27.160.62:37989: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
I20260925 01:33:48.882808 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:48.882388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:53191)
0925 01:33:48.882554 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:48.882637 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 01:33:48.883934 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:48.883452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:37211)
0925 01:33:48.883642 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:48.883759 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 01:33:48.884254 12766 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:33:49.213310 12769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:33:49.213668 12769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:33:49.213848 12769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:33:49.221015 12769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:33:49.221218 12769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:33:49.221422 12769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:33:49.231248 12769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:33397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=45663
--tserver_master_addrs=127.27.160.62:37989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:33:49.232627 12769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:33:49.234014 12769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:33:49.247576 12774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:49.247658 12775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:33:49.247869 12777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:33:49.248960 12769 server_base.cc:1034] running on GCE node
I20260925 01:33:49.249720 12769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:33:49.250973 12769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:33:49.252230 12769 hybrid_clock.cc:648] HybridClock initialized: now 1790300029252194 us; error 59 us; skew 500 ppm
I20260925 01:33:49.252580 12769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:33:49 dist-test-slave-h6hf krb5kdc[11315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:33:49.256157 12769 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:33:49.257743 12769 webserver.cc:460] Webserver started at http://127.27.160.3:45663/ using document root <none> and password file <none>
I20260925 01:33:49.258522 12769 fs_manager.cc:359] Metadata directory not provided
I20260925 01:33:49.258679 12769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:33:49.264389 12769 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:33:49.267477 12784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:49.269250 12769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:33:49.269450 12769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "87909428a991490d80bd73479305f956"
format_stamp: "Formatted at 2026-09-25 01:33:03 on dist-test-slave-h6hf"
server_key: "ba993ea6dee43634bdd5dcc7b84135e5"
server_key_iv: "38d613830114ecd494e68508ed76ebca"
server_key_version: "encryptionkey@0"
I20260925 01:33:49.270545 12769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:33:49.286983 12769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:33:49.291224 12769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:33:49.291507 12769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:33:49.292518 12769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:33:49.294034 12769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:33:49.294165 12769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:49.294304 12769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:33:49.294394 12769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:33:49.329654 12769 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:33397
I20260925 01:33:49.329752 12896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:33397 every 8 connection(s)
I20260925 01:33:49.331198 12769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:33:49.336185 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 12769
I20260925 01:33:49.336346 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:49.334290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:36339)
0925 01:33:49.335030 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:49.335267 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":271,"threads_started":1}
W20260925 01:33:49.336994 12897 heartbeater.cc:644] Failed to heartbeat to 127.27.160.62:37989 (0 consecutive failures): Network error: Failed to ping master at 127.27.160.62:37989: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
I20260925 01:33:49.337972 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:49.337579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:54141)
0925 01:33:49.337755 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:49.337836 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 01:33:49.338920 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:49.338524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:41089)
0925 01:33:49.338716 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:49.338801 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260925 01:33:49.339169 12897 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:33:49.403168 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:49.402340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:55541)
0925 01:33:49.402867 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:49.402953 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":252,"threads_started":1}
I20260925 01:33:49.885649 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:49.884824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:50747)
0925 01:33:49.885323 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:49.885425 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":220,"threads_started":1}
I20260925 01:33:50.131404 12903 raft_consensus.cc:482] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:33:50.131796 12903 raft_consensus.cc:504] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:50.133709 12903 leader_election.cc:288] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969), 87909428a991490d80bd73479305f956 (127.27.160.3:33397)
Sep 25 01:33:50 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
Sep 25 01:33:50 dist-test-slave-h6hf krb5kdc[11315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:33:50.168735 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:50.134317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:59855 (local address 127.27.160.2:41969)
0925 01:33:50.146603 (+ 12286us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:50.146621 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:50.147208 (+ 587us) server_negotiation.cc:407] Connection header received
0925 01:33:50.149617 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:50.149627 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:50.150065 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:50.150435 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:50.152476 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:50.153712 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:50.155871 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:50.156515 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:50.156738 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:50.161544 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:50.161551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:50.161564 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:50.161600 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:50.164557 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:50.165536 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:50.165548 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:50.165554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:50.165740 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:50.166194 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:50.166200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:50.166203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:50.166971 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:50.167159 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:50.167943 (+ 784us) server_negotiation.cc:299] Negotiation successful
0925 01:33:50.168313 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12097,"thread_start_us":11936,"threads_started":1}
I20260925 01:33:50.168740 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:50.134317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:34041 (local address 127.27.160.3:33397)
0925 01:33:50.146214 (+ 11897us) server_negotiation.cc:206] Beginning negotiation
0925 01:33:50.146242 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 01:33:50.147208 (+ 966us) server_negotiation.cc:407] Connection header received
0925 01:33:50.149666 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:33:50.149675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:33:50.150066 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:33:50.150436 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:33:50.152462 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:50.153737 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:50.155886 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:33:50.156531 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:33:50.156735 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:50.161326 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:33:50.161334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:33:50.161351 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:33:50.161389 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:33:50.164570 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:50.165542 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:50.165548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:50.165554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:50.165783 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:33:50.166320 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:33:50.166325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:33:50.166328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:33:50.166945 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:33:50.167138 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 01:33:50.167930 (+ 792us) server_negotiation.cc:299] Negotiation successful
0925 01:33:50.168305 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11621,"thread_start_us":11473,"threads_started":1}
I20260925 01:33:50.168768 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:50.134027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:33397 (local address 127.27.160.1:34041)
0925 01:33:50.146941 (+ 12914us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:50.147003 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:50.147788 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:50.150582 (+ 2794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:50.150593 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:50.151539 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:33:50.152270 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:50.152290 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:50.153909 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:50.153919 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:50.155734 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:50.155747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:50.156786 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:50.156793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:50.157061 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:50.157997 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:33:50.158019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:33:50.160959 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:33:50.164729 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:50.164734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:50.164746 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:50.165390 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:50.165934 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:50.165939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:50.165942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:50.166178 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:50.167104 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:33:50.167111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:33:50.167422 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 01:33:50.167693 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 01:33:50.168299 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12751,"mutex_wait_us":216,"spinlock_wait_cycles":33408,"thread_start_us":12744,"threads_started":1}
I20260925 01:33:50.168933 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:50.134025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:41969 (local address 127.27.160.1:59855)
0925 01:33:50.147057 (+ 13032us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:50.147080 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:33:50.147701 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:33:50.150582 (+ 2881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:33:50.150593 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:33:50.151534 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:33:50.152264 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:50.152284 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:50.153906 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:50.153910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:50.155721 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:33:50.155734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:33:50.156751 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:33:50.156761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:33:50.157055 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:33:50.158011 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:33:50.158028 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:33:50.161270 (+ 3242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:33:50.164719 (+ 3449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:50.164728 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:50.164744 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:50.165398 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:50.165865 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:33:50.165870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:33:50.165875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:33:50.166056 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:33:50.167106 (+ 1050us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:33:50.167112 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:33:50.167422 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 01:33:50.167693 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 01:33:50.168478 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12921,"mutex_wait_us":543,"spinlock_wait_cycles":96256,"thread_start_us":12556,"threads_started":1}
W20260925 01:33:50.172267 12521 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
W20260925 01:33:50.172521 12522 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87909428a991490d80bd73479305f956 (127.27.160.3:33397): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
I20260925 01:33:50.172683 12522 leader_election.cc:302] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b83f7a4f20d645b08667e14c09c2bb52; no voters: 87909428a991490d80bd73479305f956, baa5e7ab94744b32a03f2877a00a2323
I20260925 01:33:50.173161 12903 raft_consensus.cc:2726] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:33:50.340579 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:50.339736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:34617)
0925 01:33:50.340211 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:50.340355 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":218,"threads_started":1}
I20260925 01:33:50.404759 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:50.404317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:55577)
0925 01:33:50.404484 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:50.404573 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:33:50.887508 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:50.886724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:33577)
0925 01:33:50.887225 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:50.887335 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":281,"threads_started":1}
I20260925 01:33:51.342605 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:51.341670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:48713)
0925 01:33:51.342226 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:51.342376 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":318,"threads_started":1}
I20260925 01:33:51.406677 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:51.405868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:52697)
0925 01:33:51.406385 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:51.406483 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":244,"threads_started":1}
I20260925 01:33:51.889501 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:51.888689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:54303)
0925 01:33:51.889231 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:51.889326 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260925 01:33:52.344408 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:52.343667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:58717)
0925 01:33:52.344165 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:52.344250 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":250,"threads_started":1}
I20260925 01:33:52.360122 12914 raft_consensus.cc:482] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:33:52.360385 12914 raft_consensus.cc:504] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:52.361182 12914 leader_election.cc:288] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969), 87909428a991490d80bd73479305f956 (127.27.160.3:33397)
W20260925 01:33:52.362250 12521 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
W20260925 01:33:52.362476 12522 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87909428a991490d80bd73479305f956 (127.27.160.3:33397): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
I20260925 01:33:52.362718 12522 leader_election.cc:302] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b83f7a4f20d645b08667e14c09c2bb52; no voters: 87909428a991490d80bd73479305f956, baa5e7ab94744b32a03f2877a00a2323
I20260925 01:33:52.363049 12914 raft_consensus.cc:2726] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:33:52.408912 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:52.407875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:48191)
0925 01:33:52.408554 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:52.408655 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":368,"threads_started":1}
I20260925 01:33:52.891445 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:52.890632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:59357)
0925 01:33:52.891151 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:52.891251 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":246,"threads_started":1}
I20260925 01:33:53.346315 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:53.345517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:57623)
0925 01:33:53.346017 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:53.346104 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":272,"threads_started":1}
I20260925 01:33:53.411018 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:53.410135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:57071)
0925 01:33:53.410716 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:53.410827 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":281,"threads_started":1}
I20260925 01:33:53.893473 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:53.892585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:60715)
0925 01:33:53.893178 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:53.893292 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":269,"threads_started":1}
I20260925 01:33:54.348016 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:54.347303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:33045)
0925 01:33:54.347757 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:54.347847 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":239,"threads_started":1}
I20260925 01:33:54.412849 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:54.412062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:54923)
0925 01:33:54.412545 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:54.412644 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
I20260925 01:33:54.895423 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:54.894517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:48495)
0925 01:33:54.895080 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:54.895231 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":280,"threads_started":1}
I20260925 01:33:55.104158 12923 raft_consensus.cc:482] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:33:55.104416 12923 raft_consensus.cc:504] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:55.105171 12923 leader_election.cc:288] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969), 87909428a991490d80bd73479305f956 (127.27.160.3:33397)
W20260925 01:33:55.106292 12521 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
W20260925 01:33:55.106524 12522 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87909428a991490d80bd73479305f956 (127.27.160.3:33397): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
I20260925 01:33:55.106647 12522 leader_election.cc:302] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b83f7a4f20d645b08667e14c09c2bb52; no voters: 87909428a991490d80bd73479305f956, baa5e7ab94744b32a03f2877a00a2323
I20260925 01:33:55.106915 12923 raft_consensus.cc:2726] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:33:55.349988 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:55.349091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:34295)
0925 01:33:55.349734 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:55.349817 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":265,"threads_started":1}
I20260925 01:33:55.414863 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:55.414003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:33767)
0925 01:33:55.414518 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:55.414652 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":255,"threads_started":1}
I20260925 01:33:55.897485 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:55.896483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:43159)
0925 01:33:55.897102 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:55.897284 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":269,"threads_started":1}
I20260925 01:33:56.351859 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:56.351048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:47581)
0925 01:33:56.351568 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:56.351658 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":238,"threads_started":1}
I20260925 01:33:56.416747 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:56.415954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:52903)
0925 01:33:56.416462 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:56.416559 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":270,"threads_started":1}
I20260925 01:33:56.899228 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:56.898489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:44503)
0925 01:33:56.898947 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:56.899044 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":241,"threads_started":1}
I20260925 01:33:57.129047 12930 raft_consensus.cc:482] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:33:57.129374 12930 raft_consensus.cc:504] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:33:57.130190 12930 leader_election.cc:288] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969), 87909428a991490d80bd73479305f956 (127.27.160.3:33397)
W20260925 01:33:57.131254 12521 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
W20260925 01:33:57.131486 12522 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87909428a991490d80bd73479305f956 (127.27.160.3:33397): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
I20260925 01:33:57.131626 12522 leader_election.cc:302] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b83f7a4f20d645b08667e14c09c2bb52; no voters: 87909428a991490d80bd73479305f956, baa5e7ab94744b32a03f2877a00a2323
I20260925 01:33:57.131944 12930 raft_consensus.cc:2726] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:33:57.353796 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:57.352948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:38983)
0925 01:33:57.353521 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:57.353606 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":362,"threads_started":1}
I20260925 01:33:57.418869 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:57.417901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:60815)
0925 01:33:57.418485 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:57.418622 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":303,"threads_started":1}
I20260925 01:33:57.901311 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:57.900416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:49185)
0925 01:33:57.900935 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:57.901037 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":266,"threads_started":1}
I20260925 01:33:58.355703 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:58.354867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:53803)
0925 01:33:58.355378 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:58.355479 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":254,"threads_started":1}
I20260925 01:33:58.421006 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:58.420236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:40677)
0925 01:33:58.420737 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:58.420831 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":246,"threads_started":1}
I20260925 01:33:58.903033 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:58.902298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:39991)
0925 01:33:58.902763 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:58.902853 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260925 01:33:59.357609 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:59.356763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:37689)
0925 01:33:59.357269 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:59.357383 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":225,"threads_started":1}
I20260925 01:33:59.422901 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:59.422060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:49445)
0925 01:33:59.422631 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:59.422725 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":324,"threads_started":1}
I20260925 01:33:59.904939 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:33:59.904079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:40369)
0925 01:33:59.904639 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 01:33:59.904749 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":289,"threads_started":1}
I20260925 01:34:00.359773 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:00.358855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:40033)
0925 01:34:00.359515 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:00.359602 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260925 01:34:00.425247 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:00.424194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:59177)
0925 01:34:00.424809 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:00.424978 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":259,"threads_started":1}
I20260925 01:34:00.906783 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:00.905971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:34669)
0925 01:34:00.906430 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:00.906525 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
I20260925 01:34:01.361718 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:01.360784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:59885)
0925 01:34:01.361352 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:01.361464 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":257,"threads_started":1}
I20260925 01:34:01.427193 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:01.426392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:37777)
0925 01:34:01.426886 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:01.426970 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":262,"threads_started":1}
I20260925 01:34:01.908839 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:01.907902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:50655)
0925 01:34:01.908452 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:01.908576 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":271,"threads_started":1}
I20260925 01:34:02.363749 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:02.362776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:47721)
0925 01:34:02.363350 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:02.363492 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":291,"threads_started":1}
I20260925 01:34:02.429282 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:02.428351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:45323)
0925 01:34:02.428861 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:02.428975 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":250,"threads_started":1}
I20260925 01:34:02.910683 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:02.909835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:49863)
0925 01:34:02.910393 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:02.910485 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":330,"threads_started":1}
I20260925 01:34:03.365842 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:03.364850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:33675)
0925 01:34:03.365504 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:03.365621 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":386,"threads_started":1}
I20260925 01:34:03.431529 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:03.430692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.1:34271)
0925 01:34:03.431247 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:03.431336 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":274,"threads_started":1}
I20260925 01:34:03.912858 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:03.912004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.2:55613)
0925 01:34:03.912538 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:03.912636 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":250,"threads_started":1}
I20260925 01:34:04.049522 12952 raft_consensus.cc:482] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:34:04.049729 12952 raft_consensus.cc:504] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "baa5e7ab94744b32a03f2877a00a2323" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 41969 } } peers { permanent_uuid: "b83f7a4f20d645b08667e14c09c2bb52" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40009 } } peers { permanent_uuid: "87909428a991490d80bd73479305f956" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33397 } }
I20260925 01:34:04.050419 12952 leader_election.cc:288] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969), 87909428a991490d80bd73479305f956 (127.27.160.3:33397)
W20260925 01:34:04.051451 12522 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87909428a991490d80bd73479305f956 (127.27.160.3:33397): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
W20260925 01:34:04.051641 12521 leader_election.cc:341] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer baa5e7ab94744b32a03f2877a00a2323 (127.27.160.2:41969): Not found: Tablet not found: 417bd3bbadce4b1a8e0afd12c5dd87a8
I20260925 01:34:04.051744 12521 leader_election.cc:302] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b83f7a4f20d645b08667e14c09c2bb52; no voters: 87909428a991490d80bd73479305f956, baa5e7ab94744b32a03f2877a00a2323
I20260925 01:34:04.051982 12952 raft_consensus.cc:2726] T 417bd3bbadce4b1a8e0afd12c5dd87a8 P b83f7a4f20d645b08667e14c09c2bb52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:34:04.338701 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 12464
I20260925 01:34:04.356061 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:04.340016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.27.160.1:40009)
0925 01:34:04.340744 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:04.340755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:04.340795 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:34:04.340890 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:04.340904 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:04.341267 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:04.341465 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:04.343534 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:04.344753 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:04.346253 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:04.346947 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:04.347076 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:04.350105 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:04.350118 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:04.350134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:04.350171 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:04.353341 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:04.354059 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:04.354066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:04.354069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:04.354215 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:04.354585 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:04.354588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:04.354590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:04.354947 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:04.355178 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:04.355547 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 01:34:04.355780 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":321,"threads_started":1}
I20260925 01:34:04.357486 12630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:04.368013 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:04.367184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:37989 (local address 127.27.160.3:59751)
0925 01:34:04.367718 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:04.367857 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.62:37989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":294,"threads_started":1}
I20260925 01:34:04.454118 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 12464
I20260925 01:34:04.475425 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 12638
I20260925 01:34:04.490566 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:04.476250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.27.160.2:41969)
0925 01:34:04.476805 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:04.476810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:04.476827 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:34:04.476888 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:04.476892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:04.476948 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:04.477043 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:04.478493 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:04.479342 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:04.480782 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:04.481373 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:04.481487 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:04.484264 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:04.484269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:04.484273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:04.484310 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:04.487335 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:04.488246 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:04.488250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:04.488253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:04.488413 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:04.488813 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:04.488817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:04.488819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:04.489187 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:04.489433 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:04.489895 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 01:34:04.490083 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":345,"threads_started":1}
I20260925 01:34:04.491364 12761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:04.578871 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 12638
I20260925 01:34:04.597295 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 12769
I20260925 01:34:04.612672 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:04.598228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.27.160.3:33397)
0925 01:34:04.598814 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:04.598819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:04.598837 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:34:04.598891 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:04.598895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:04.598969 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:04.599067 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:04.600862 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:04.601768 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:04.603298 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:04.603782 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:04.603904 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:04.606927 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:04.606933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:04.606936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:04.606973 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:04.610021 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:04.610867 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:04.610872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:04.610874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:04.611038 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:04.611457 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:04.611461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:04.611463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:04.611739 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:04.611938 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:04.612202 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 01:34:04.612376 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":312,"threads_started":1}
I20260925 01:34:04.613582 12892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:04.699010 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 12769
2026-09-25T01:34:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64916 ms)
[----------] 30 tests from SecurityITest (307132 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:04 dist-test-slave-h6hf krb5kdc[12963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:04 dist-test-slave-h6hf krb5kdc[12963](info): set up 2 sockets
Sep 25 01:34:04 dist-test-slave-h6hf krb5kdc[12963](info): commencing operation
krb5kdc: starting...
W20260925 01:34:05.890810 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:06 dist-test-slave-h6hf krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:34:06.169413 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:41577
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:41577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:06.536576 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:06.536926 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:06.537020 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:06.544864 12976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:06.545003 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:06.545053 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:06.545181 12976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:06.545277 12976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:06.555857 12976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:41577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:06.557399 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:06.558933 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:06.572266 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:06.572378 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:06.572355 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:06.573204 12976 server_base.cc:1034] running on GCE node
I20260925 01:34:06.574159 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:06.575850 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:06.577106 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790300046577030 us; error 73 us; skew 500 ppm
I20260925 01:34:06.577564 12976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:06 dist-test-slave-h6hf krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300046, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:06.581001 12976 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:34:06.582791 12976 webserver.cc:460] Webserver started at http://127.27.160.62:37949/ using document root <none> and password file <none>
I20260925 01:34:06.583575 12976 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:06.583739 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:06.584089 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:06.587595 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "fcba12fdc3e645c199bc165dda5d5007"
format_stamp: "Formatted at 2026-09-25 01:34:06 on dist-test-slave-h6hf"
server_key: "2515fc00876e5e0d6df523b377710cbe"
server_key_iv: "b05d9528f723fba8419b52fa8e38595e"
server_key_version: "encryptionkey@0"
I20260925 01:34:06.588472 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "fcba12fdc3e645c199bc165dda5d5007"
format_stamp: "Formatted at 2026-09-25 01:34:06 on dist-test-slave-h6hf"
server_key: "2515fc00876e5e0d6df523b377710cbe"
server_key_iv: "b05d9528f723fba8419b52fa8e38595e"
server_key_version: "encryptionkey@0"
I20260925 01:34:06.594347 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:34:06.598306 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:06.599872 12976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260925 01:34:06.600088 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "fcba12fdc3e645c199bc165dda5d5007"
format_stamp: "Formatted at 2026-09-25 01:34:06 on dist-test-slave-h6hf"
server_key: "2515fc00876e5e0d6df523b377710cbe"
server_key_iv: "b05d9528f723fba8419b52fa8e38595e"
server_key_version: "encryptionkey@0"
I20260925 01:34:06.600319 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:06.622141 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:06.627372 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:06.627713 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:06.648623 12976 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:41577
I20260925 01:34:06.648682 13042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:41577 every 8 connection(s)
I20260925 01:34:06.650216 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:06.654798 13043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:06.657927 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 12976
I20260925 01:34:06.658178 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:06.658912 28288 external_mini_cluster.cc:1428] Setting key 0f3fd62aad44742747df09995d5b2694
I20260925 01:34:06.668064 13043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007: Bootstrap starting.
I20260925 01:34:06.671980 13043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:06.673260 13043 log.cc:824] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:34:06 dist-test-slave-h6hf krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:06.677222 13043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007: No bootstrap required, opened a new log
I20260925 01:34:06.685420 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:06.661265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.27.160.62:41577)
0925 01:34:06.662175 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:06.662189 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:06.662234 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:34:06.664071 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:06.664097 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:06.664564 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:06.665018 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:06.666666 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:06.667997 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:06.669611 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:06.670368 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:06.670583 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:06.675550 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:06.675562 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:06.675584 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:06.675634 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:06.679986 (+ 4352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:06.681012 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:06.681022 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:06.681032 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:06.681293 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:06.681793 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:06.681799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:06.681803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:06.682592 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:06.682909 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:06.683404 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 01:34:06.683892 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":405,"threads_started":1}
I20260925 01:34:06.688090 13043 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcba12fdc3e645c199bc165dda5d5007" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41577 } }
I20260925 01:34:06.688572 13043 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:06.688699 13043 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcba12fdc3e645c199bc165dda5d5007, State: Initialized, Role: FOLLOWER
I20260925 01:34:06.689383 13043 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcba12fdc3e645c199bc165dda5d5007" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41577 } }
I20260925 01:34:06.689654 13043 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:06.689801 13043 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:06.689985 13043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:06.692483 13043 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcba12fdc3e645c199bc165dda5d5007" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41577 } }
I20260925 01:34:06.693039 13043 leader_election.cc:302] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcba12fdc3e645c199bc165dda5d5007; no voters:
I20260925 01:34:06.693837 13043 leader_election.cc:288] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:06.694031 13048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:06.694916 13048 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [term 1 LEADER]: Becoming Leader. State: Replica: fcba12fdc3e645c199bc165dda5d5007, State: Running, Role: LEADER
I20260925 01:34:06.695566 13048 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcba12fdc3e645c199bc165dda5d5007" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41577 } }
I20260925 01:34:06.696273 13043 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:06.699167 13049 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcba12fdc3e645c199bc165dda5d5007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcba12fdc3e645c199bc165dda5d5007" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41577 } } }
I20260925 01:34:06.699167 13050 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcba12fdc3e645c199bc165dda5d5007. Latest consensus state: current_term: 1 leader_uuid: "fcba12fdc3e645c199bc165dda5d5007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcba12fdc3e645c199bc165dda5d5007" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41577 } } }
I20260925 01:34:06.699723 13050 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:06.699719 13049 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:06.705830 13056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:06.711927 13056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:06.721866 13056 catalog_manager.cc:1347] Generated new cluster ID: 21b3a35f2cd44b6aa8b5a2bee8e018f7
I20260925 01:34:06.722009 13056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:06.762302 13056 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:06.763680 13056 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:06.776566 13056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007: Generated new TSK 0
I20260925 01:34:06.777712 13056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:34:06.851563 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41577
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:07.218128 13069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:07.218433 13069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:07.218601 13069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:07.226066 13069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:07.226195 13069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:07.226393 13069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:34:07.236359 13069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:07.237787 13069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:07.239176 13069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:07.253346 13074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:07.253358 13077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:07.253422 13075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:07.254565 13069 server_base.cc:1034] running on GCE node
I20260925 01:34:07.255308 13069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:07.256322 13069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:07.257573 13069 hybrid_clock.cc:648] HybridClock initialized: now 1790300047257494 us; error 55 us; skew 500 ppm
I20260925 01:34:07.257932 13069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:07 dist-test-slave-h6hf krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300047, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:07.261427 13069 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:34:07.262985 13069 webserver.cc:460] Webserver started at http://127.27.160.1:41471/ using document root <none> and password file <none>
I20260925 01:34:07.263744 13069 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:07.263899 13069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:07.264241 13069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:07.267335 13069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "9d2d95d6d52b41f59758a5469482621d"
format_stamp: "Formatted at 2026-09-25 01:34:07 on dist-test-slave-h6hf"
server_key: "9ebbd318a78018c112590f6a3c30af00"
server_key_iv: "7702ce2cf7f9c80aab0c8e1e75241ffa"
server_key_version: "encryptionkey@0"
I20260925 01:34:07.268451 13069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "9d2d95d6d52b41f59758a5469482621d"
format_stamp: "Formatted at 2026-09-25 01:34:07 on dist-test-slave-h6hf"
server_key: "9ebbd318a78018c112590f6a3c30af00"
server_key_iv: "7702ce2cf7f9c80aab0c8e1e75241ffa"
server_key_version: "encryptionkey@0"
I20260925 01:34:07.274691 13069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260925 01:34:07.279248 13084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:07.281018 13069 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 01:34:07.281267 13069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "9d2d95d6d52b41f59758a5469482621d"
format_stamp: "Formatted at 2026-09-25 01:34:07 on dist-test-slave-h6hf"
server_key: "9ebbd318a78018c112590f6a3c30af00"
server_key_iv: "7702ce2cf7f9c80aab0c8e1e75241ffa"
server_key_version: "encryptionkey@0"
I20260925 01:34:07.281474 13069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:07.306417 13069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:07.311179 13069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:07.311429 13069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:07.312362 13069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:07.313741 13069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:07.313815 13069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:07.313902 13069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:07.313938 13069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:07.348639 13069 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:43867
I20260925 01:34:07.348766 13196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:43867 every 8 connection(s)
I20260925 01:34:07.350124 13069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:34:07.352486 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 13069
I20260925 01:34:07.352710 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:34:07.353646 28288 external_mini_cluster.cc:1428] Setting key b491f9328daa32eb38732540161a852a
Sep 25 01:34:07 dist-test-slave-h6hf krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300047, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:07.376278 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:07.353238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:60133 (local address 127.27.160.62:41577)
0925 01:34:07.353805 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:07.353812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:07.355051 (+ 1239us) server_negotiation.cc:407] Connection header received
0925 01:34:07.357725 (+ 2674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:07.357730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:07.357801 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:07.357902 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:07.360183 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:07.361007 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:07.363658 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:07.364120 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:07.364259 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:07.368359 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:07.368371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:07.368380 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:07.368432 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:07.372499 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:07.373443 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:07.373453 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:07.373461 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:07.373644 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:07.374085 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:07.374094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:07.374100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:07.374515 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:07.374709 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:07.375408 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 01:34:07.375735 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":284,"threads_started":1}
I20260925 01:34:07.376850 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:07.353974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41577 (local address 127.27.160.1:60133)
0925 01:34:07.354893 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:07.354940 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:07.357398 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:07.358188 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:07.358203 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:07.359061 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:07.359956 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:07.359985 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:07.361223 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:07.361231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:07.363482 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:07.363501 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:07.364394 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:07.364403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:07.364758 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:07.365596 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:07.365633 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:07.368088 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:07.372678 (+ 4590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:07.372687 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:07.372703 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:07.373275 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:07.373768 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:07.373771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:07.373773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:07.373960 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:07.374652 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:07.374659 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:07.374997 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 01:34:07.375239 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 01:34:07.375644 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":351,"threads_started":1}
I20260925 01:34:07.378790 13197 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41577
I20260925 01:34:07.379169 13197 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:07.379989 13197 heartbeater.cc:511] Master 127.27.160.62:41577 requested a full tablet report, sending...
I20260925 01:34:07.382658 13008 ts_manager.cc:194] Registered new tserver with Master: 9d2d95d6d52b41f59758a5469482621d (127.27.160.1:43867)
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
I20260925 01:34:07.384322 13008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:60133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:34:07.421013 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41577
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:07.762774 13203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:07.763078 13203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:07.763173 13203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:07.770296 13203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:07.770412 13203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:07.770529 13203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:34:07.780673 13203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:07.782019 13203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:07.783262 13203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:07.797541 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:07.797451 13211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:07.797482 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:07.798362 13203 server_base.cc:1034] running on GCE node
I20260925 01:34:07.799048 13203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:07.800055 13203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:07.801221 13203 hybrid_clock.cc:648] HybridClock initialized: now 1790300047801168 us; error 44 us; skew 500 ppm
I20260925 01:34:07.801558 13203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:07 dist-test-slave-h6hf krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300047, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:07.804939 13203 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:34:07.806579 13203 webserver.cc:460] Webserver started at http://127.27.160.2:34001/ using document root <none> and password file <none>
I20260925 01:34:07.807312 13203 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:07.807448 13203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:07.807832 13203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:07.811041 13203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "74c06fb89f904c2db00a9f0ef6babbbd"
format_stamp: "Formatted at 2026-09-25 01:34:07 on dist-test-slave-h6hf"
server_key: "61891ca58c9eead816ffca85473cc21b"
server_key_iv: "0dea0b0233ea246183e374230032f78b"
server_key_version: "encryptionkey@0"
I20260925 01:34:07.811852 13203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "74c06fb89f904c2db00a9f0ef6babbbd"
format_stamp: "Formatted at 2026-09-25 01:34:07 on dist-test-slave-h6hf"
server_key: "61891ca58c9eead816ffca85473cc21b"
server_key_iv: "0dea0b0233ea246183e374230032f78b"
server_key_version: "encryptionkey@0"
I20260925 01:34:07.817453 13203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:34:07.821224 13218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:07.822723 13203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:07.822916 13203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "74c06fb89f904c2db00a9f0ef6babbbd"
format_stamp: "Formatted at 2026-09-25 01:34:07 on dist-test-slave-h6hf"
server_key: "61891ca58c9eead816ffca85473cc21b"
server_key_iv: "0dea0b0233ea246183e374230032f78b"
server_key_version: "encryptionkey@0"
I20260925 01:34:07.823154 13203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:07.839506 13203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:07.844206 13203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:07.844521 13203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:07.845654 13203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:07.847146 13203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:07.847227 13203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:07.847352 13203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:07.847414 13203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:07.882820 13203 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:43281
I20260925 01:34:07.882894 13330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:43281 every 8 connection(s)
I20260925 01:34:07.884234 13203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:34:07.888012 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 13203
I20260925 01:34:07.888288 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:34:07.889047 28288 external_mini_cluster.cc:1428] Setting key 4ba3368fa6b4c0f23cd5e0af6d16e831
Sep 25 01:34:07 dist-test-slave-h6hf krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300047, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:07.910727 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:07.887252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:50591 (local address 127.27.160.62:41577)
0925 01:34:07.887728 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:07.887735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:07.888913 (+ 1178us) server_negotiation.cc:407] Connection header received
0925 01:34:07.891476 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:07.891480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:07.891544 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:07.891634 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:07.894120 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:07.894814 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:07.897633 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:07.898157 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:07.898279 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:07.903028 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:07.903035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:07.903040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:07.903081 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:07.906781 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:07.907612 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:07.907621 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:07.907628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:07.907804 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:07.908391 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:07.908399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:07.908405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:07.908841 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:07.909008 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:07.909895 (+ 887us) server_negotiation.cc:299] Negotiation successful
0925 01:34:07.910196 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":212,"threads_started":1}
I20260925 01:34:07.911342 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:07.887719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41577 (local address 127.27.160.2:50591)
0925 01:34:07.888695 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:07.888759 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:07.891132 (+ 2373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:07.891977 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:07.891993 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:07.892900 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:07.893920 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:07.893951 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:07.894980 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:07.894989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:07.897442 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:07.897464 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:07.898430 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:07.898441 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:07.898819 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:07.899643 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:07.899680 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:07.902728 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:07.906980 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:07.906988 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:07.907002 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:07.907460 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:07.907966 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:07.907977 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:07.907981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:07.908230 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:07.909011 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:07.909021 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:07.909483 (+ 462us) client_negotiation.cc:769] Sending connection context
0925 01:34:07.909716 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 01:34:07.910139 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":336,"threads_started":1}
I20260925 01:34:07.912902 13331 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41577
I20260925 01:34:07.913321 13331 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:07.914095 13331 heartbeater.cc:511] Master 127.27.160.62:41577 requested a full tablet report, sending...
I20260925 01:34:07.915693 13008 ts_manager.cc:194] Registered new tserver with Master: 74c06fb89f904c2db00a9f0ef6babbbd (127.27.160.2:43281)
I20260925 01:34:07.916543 13008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:50591
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:34:07.957690 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41577
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:42059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:08.316015 13337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:08.316386 13337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:08.316563 13337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:08.324499 13337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:08.324781 13337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:08.324988 13337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:34:08.335407 13337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:42059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:08.336858 13337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:08.338369 13337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:08.352852 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:08.353050 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:08.352885 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:08.355067 13337 server_base.cc:1034] running on GCE node
I20260925 01:34:08.355780 13337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:08.356848 13337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:08.358176 13337 hybrid_clock.cc:648] HybridClock initialized: now 1790300048358050 us; error 87 us; skew 500 ppm
I20260925 01:34:08.358580 13337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:08 dist-test-slave-h6hf krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:08.362422 13337 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:34:08.364138 13337 webserver.cc:460] Webserver started at http://127.27.160.3:33797/ using document root <none> and password file <none>
I20260925 01:34:08.364966 13337 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:08.365188 13337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:08.365576 13337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:08.368891 13337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "d14e47c24546435a907a918c15772ff2"
format_stamp: "Formatted at 2026-09-25 01:34:08 on dist-test-slave-h6hf"
server_key: "3bc7e05da4a2c185a7d7b483976c405b"
server_key_iv: "50654226856bdd410a5e5477ff22208e"
server_key_version: "encryptionkey@0"
I20260925 01:34:08.369900 13337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "d14e47c24546435a907a918c15772ff2"
format_stamp: "Formatted at 2026-09-25 01:34:08 on dist-test-slave-h6hf"
server_key: "3bc7e05da4a2c185a7d7b483976c405b"
server_key_iv: "50654226856bdd410a5e5477ff22208e"
server_key_version: "encryptionkey@0"
I20260925 01:34:08.375983 13337 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 01:34:08.380121 13352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:08.381882 13337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:08.382102 13337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "d14e47c24546435a907a918c15772ff2"
format_stamp: "Formatted at 2026-09-25 01:34:08 on dist-test-slave-h6hf"
server_key: "3bc7e05da4a2c185a7d7b483976c405b"
server_key_iv: "50654226856bdd410a5e5477ff22208e"
server_key_version: "encryptionkey@0"
I20260925 01:34:08.382354 13337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:08.388281 13197 heartbeater.cc:503] Master 127.27.160.62:41577 was elected leader, sending a full tablet report...
I20260925 01:34:08.432021 13337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:08.437351 13337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:08.437789 13337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:08.438964 13337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:08.440542 13337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:08.440670 13337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:08.440833 13337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:08.440917 13337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:08.479545 13337 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:40641
I20260925 01:34:08.479583 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:40641 every 8 connection(s)
I20260925 01:34:08.481036 13337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:34:08.487481 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 13337
I20260925 01:34:08.487762 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:34:08.488274 28288 external_mini_cluster.cc:1428] Setting key 11edca778e88ebaf8dfd9ea9bd466a71
Sep 25 01:34:08 dist-test-slave-h6hf krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300048, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:08.503479 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.484158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:60083 (local address 127.27.160.62:41577)
0925 01:34:08.484662 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.484667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.485839 (+ 1172us) server_negotiation.cc:407] Connection header received
0925 01:34:08.488104 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.488110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.488208 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.488334 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:08.489952 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.490782 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.492372 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.492788 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.492922 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.496520 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:08.496525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:08.496529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:08.496585 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:08.499649 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:08.500562 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:08.500569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:08.500573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:08.500731 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:08.501319 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:08.501323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:08.501325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:08.501622 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:08.501751 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.502774 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.503043 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":262,"threads_started":1}
I20260925 01:34:08.504808 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.484803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41577 (local address 127.27.160.3:60083)
0925 01:34:08.485658 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:08.485704 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:08.487791 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:08.488592 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:08.488600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:08.489165 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:08.489777 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.489795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.490934 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.490939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.492212 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.492225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.493023 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.493031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.493345 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.494024 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:08.494046 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:08.496323 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:08.499825 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:08.499833 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:08.499848 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:08.500381 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:08.500862 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:08.500866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:08.500869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:08.501103 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:08.501807 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:08.501816 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:08.502318 (+ 502us) client_negotiation.cc:769] Sending connection context
0925 01:34:08.502659 (+ 341us) client_negotiation.cc:240] Negotiation successful
0925 01:34:08.503280 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":344,"threads_started":1}
I20260925 01:34:08.506412 13465 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41577
I20260925 01:34:08.506803 13465 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:08.507615 13465 heartbeater.cc:511] Master 127.27.160.62:41577 requested a full tablet report, sending...
I20260925 01:34:08.509338 13008 ts_manager.cc:194] Registered new tserver with Master: d14e47c24546435a907a918c15772ff2 (127.27.160.3:40641)
I20260925 01:34:08.510334 13008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:60083
I20260925 01:34:08.517066 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:34:08.538014 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.522561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.27.160.62:41577)
0925 01:34:08.522810 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.522815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.523296 (+ 481us) server_negotiation.cc:407] Connection header received
0925 01:34:08.523435 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.523439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.523516 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.523696 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:08.525512 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.526463 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.528032 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.528514 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.528696 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.531452 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:08.531458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:08.531462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:08.531502 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:08.535061 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:08.535844 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:08.535849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:08.535851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:08.536008 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:08.536420 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:08.536424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:08.536426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:08.536853 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:08.537066 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.537487 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.537761 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:34:08.543143 13008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:34:08.551123 13008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:34:08.590364 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.579631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.27.160.3:40641)
0925 01:34:08.580391 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.580397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.581237 (+ 840us) server_negotiation.cc:407] Connection header received
0925 01:34:08.581369 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.581376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.581574 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.581749 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:08.582869 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.584006 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.587269 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.588532 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.588681 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.588937 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.589772 (+ 835us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.589970 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":441,"threads_started":1}
I20260925 01:34:08.590405 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.579002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:40641 (local address 127.0.0.1:52100)
0925 01:34:08.581090 (+ 2088us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:08.581105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:08.581249 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:08.581999 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:08.582004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:08.582042 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:08.582602 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.582618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.584157 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.584166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.587044 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.587065 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.589153 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.589161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.589507 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.589552 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 01:34:08.589720 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 01:34:08.589951 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1972,"mutex_wait_us":80,"spinlock_wait_cycles":3456,"thread_start_us":1798,"threads_started":1}
I20260925 01:34:08.590461 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.565643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.27.160.1:43867)
0925 01:34:08.579027 (+ 13384us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.579042 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.580817 (+ 1775us) server_negotiation.cc:407] Connection header received
0925 01:34:08.581206 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.581216 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.581464 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.581709 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:08.582427 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.584127 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.586983 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.588552 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.588707 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.588966 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.589935 (+ 969us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.590113 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13081,"thread_start_us":12893,"threads_started":1}
I20260925 01:34:08.590440 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.565034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.27.160.2:43281)
0925 01:34:08.579499 (+ 14465us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.579507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.580595 (+ 1088us) server_negotiation.cc:407] Connection header received
0925 01:34:08.581021 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.581031 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.581373 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.581518 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:08.582972 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.584469 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.587220 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.588604 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.588753 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.588930 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.589938 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.590129 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14319,"thread_start_us":14223,"threads_started":1}
I20260925 01:34:08.591240 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.564916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:43867 (local address 127.0.0.1:51372)
0925 01:34:08.580637 (+ 15721us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:08.580662 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:08.580938 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:08.581714 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:08.581733 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:08.581768 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:08.582200 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.582217 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.584321 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.584326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.586809 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.586845 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.589210 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.589217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.589658 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.589738 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 01:34:08.590700 (+ 962us) client_negotiation.cc:240] Negotiation successful
0925 01:34:08.590848 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15540,"mutex_wait_us":297,"thread_start_us":12827,"threads_started":1}
I20260925 01:34:08.591215 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.564821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:43281 (local address 127.0.0.1:56946)
0925 01:34:08.579078 (+ 14257us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:08.580442 (+ 1364us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:08.580706 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:08.582200 (+ 1494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:08.582205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:08.582260 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:08.582785 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.582798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.584620 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.584634 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.587076 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.587085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.589192 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.589201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.589658 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.589748 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 01:34:08.589834 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:34:08.590739 (+ 905us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13997,"mutex_wait_us":906,"spinlock_wait_cycles":2304,"thread_start_us":13066,"threads_started":1}
I20260925 01:34:08.594832 13266 tablet_service.cc:1467] Processing CreateTablet for tablet 229976d99b3f4904acb5bfecb7a007eb (DEFAULT_TABLE table=test-table [id=baa02329291f483fbd157bbf618c3ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:08.595578 13132 tablet_service.cc:1467] Processing CreateTablet for tablet 229976d99b3f4904acb5bfecb7a007eb (DEFAULT_TABLE table=test-table [id=baa02329291f483fbd157bbf618c3ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:08.595130 13400 tablet_service.cc:1467] Processing CreateTablet for tablet 229976d99b3f4904acb5bfecb7a007eb (DEFAULT_TABLE table=test-table [id=baa02329291f483fbd157bbf618c3ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:08.596235 13266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229976d99b3f4904acb5bfecb7a007eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:08.596916 13400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229976d99b3f4904acb5bfecb7a007eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:08.596915 13132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 229976d99b3f4904acb5bfecb7a007eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:08.607981 13484 tablet_bootstrap.cc:492] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Bootstrap starting.
I20260925 01:34:08.612581 13484 tablet_bootstrap.cc:654] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:08.612798 13485 tablet_bootstrap.cc:492] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Bootstrap starting.
I20260925 01:34:08.614197 13484 log.cc:824] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:08.615731 13486 tablet_bootstrap.cc:492] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Bootstrap starting.
I20260925 01:34:08.616962 13485 tablet_bootstrap.cc:654] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:08.617664 13484 tablet_bootstrap.cc:492] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: No bootstrap required, opened a new log
I20260925 01:34:08.617957 13484 ts_tablet_manager.cc:1397] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260925 01:34:08.618356 13485 log.cc:824] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:08.619813 13486 tablet_bootstrap.cc:654] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:08.621541 13486 log.cc:824] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:08.626752 13484 raft_consensus.cc:348] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.627238 13484 raft_consensus.cc:374] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:08.627337 13484 raft_consensus.cc:729] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c06fb89f904c2db00a9f0ef6babbbd, State: Initialized, Role: FOLLOWER
I20260925 01:34:08.627977 13485 tablet_bootstrap.cc:492] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: No bootstrap required, opened a new log
I20260925 01:34:08.628019 13484 consensus_queue.cc:260] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.628319 13485 ts_tablet_manager.cc:1397] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260925 01:34:08.628567 13486 tablet_bootstrap.cc:492] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: No bootstrap required, opened a new log
I20260925 01:34:08.628867 13486 ts_tablet_manager.cc:1397] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260925 01:34:08.629421 13331 heartbeater.cc:503] Master 127.27.160.62:41577 was elected leader, sending a full tablet report...
I20260925 01:34:08.629945 13484 ts_tablet_manager.cc:1428] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260925 01:34:08.636300 13332 tablet.cc:2369] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:34:08.640519 13485 raft_consensus.cc:348] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.641072 13485 raft_consensus.cc:374] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:08.641229 13485 raft_consensus.cc:729] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d14e47c24546435a907a918c15772ff2, State: Initialized, Role: FOLLOWER
I20260925 01:34:08.641188 13486 raft_consensus.cc:348] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.641647 13486 raft_consensus.cc:374] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:08.641758 13486 raft_consensus.cc:729] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2d95d6d52b41f59758a5469482621d, State: Initialized, Role: FOLLOWER
I20260925 01:34:08.641985 13485 consensus_queue.cc:260] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.642446 13486 consensus_queue.cc:260] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.643363 13465 heartbeater.cc:503] Master 127.27.160.62:41577 was elected leader, sending a full tablet report...
I20260925 01:34:08.644366 13486 ts_tablet_manager.cc:1428] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260925 01:34:08.645586 13485 ts_tablet_manager.cc:1428] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
W20260925 01:34:08.732980 13466 tablet.cc:2369] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:34:08.797967 13492 raft_consensus.cc:482] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:34:08.798259 13492 raft_consensus.cc:504] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.799997 13492 leader_election.cc:288] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c06fb89f904c2db00a9f0ef6babbbd (127.27.160.2:43281), d14e47c24546435a907a918c15772ff2 (127.27.160.3:40641)
I20260925 01:34:08.806921 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.800557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:53295 (local address 127.27.160.2:43281)
0925 01:34:08.800755 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.800759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.801271 (+ 512us) server_negotiation.cc:407] Connection header received
0925 01:34:08.801492 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.801497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.801585 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.801758 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:08.802326 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.803026 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.805062 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.805966 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.806149 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.806228 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.806510 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.806637 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:34:08.807071 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.800378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:43281 (local address 127.27.160.1:53295)
0925 01:34:08.801161 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:08.801182 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:08.801336 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:08.801790 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:08.801795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:08.801828 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:08.802169 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.802178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.803156 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.803163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.804906 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.804926 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.806176 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.806182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.806348 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.806372 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:34:08.806482 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 01:34:08.806617 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":33,"thread_start_us":434,"threads_started":1}
I20260925 01:34:08.807578 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.801474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:36587 (local address 127.27.160.3:40641)
0925 01:34:08.801692 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.801697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.802057 (+ 360us) server_negotiation.cc:407] Connection header received
0925 01:34:08.802263 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.802268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.802326 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.802426 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:08.802995 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.803802 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.805746 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.806538 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.806737 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.806815 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.807177 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.807307 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:34:08.808369 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.801324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:40641 (local address 127.27.160.1:36587)
0925 01:34:08.801945 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:08.801959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:08.802116 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:08.802441 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:08.802445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:08.802467 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:08.802841 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.802850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.803948 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.803960 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.805577 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:08.805589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.806778 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.806786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:08.806950 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.806973 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:34:08.807796 (+ 823us) client_negotiation.cc:240] Negotiation successful
0925 01:34:08.807954 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":21,"thread_start_us":313,"threads_started":1}
I20260925 01:34:08.808473 13286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229976d99b3f4904acb5bfecb7a007eb" candidate_uuid: "9d2d95d6d52b41f59758a5469482621d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" is_pre_election: true
I20260925 01:34:08.809233 13286 raft_consensus.cc:2454] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2d95d6d52b41f59758a5469482621d in term 0.
I20260925 01:34:08.809542 13420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229976d99b3f4904acb5bfecb7a007eb" candidate_uuid: "9d2d95d6d52b41f59758a5469482621d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d14e47c24546435a907a918c15772ff2" is_pre_election: true
I20260925 01:34:08.810081 13086 leader_election.cc:302] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c06fb89f904c2db00a9f0ef6babbbd, 9d2d95d6d52b41f59758a5469482621d; no voters:
I20260925 01:34:08.810159 13420 raft_consensus.cc:2454] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d2d95d6d52b41f59758a5469482621d in term 0.
I20260925 01:34:08.810564 13492 raft_consensus.cc:2781] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:34:08.810746 13492 raft_consensus.cc:482] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:34:08.810881 13492 raft_consensus.cc:3037] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:08.813177 13492 raft_consensus.cc:504] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.813838 13492 leader_election.cc:288] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [CANDIDATE]: Term 1 election: Requested vote from peers 74c06fb89f904c2db00a9f0ef6babbbd (127.27.160.2:43281), d14e47c24546435a907a918c15772ff2 (127.27.160.3:40641)
I20260925 01:34:08.814391 13286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229976d99b3f4904acb5bfecb7a007eb" candidate_uuid: "9d2d95d6d52b41f59758a5469482621d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c06fb89f904c2db00a9f0ef6babbbd"
I20260925 01:34:08.814617 13286 raft_consensus.cc:3037] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:08.814865 13420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "229976d99b3f4904acb5bfecb7a007eb" candidate_uuid: "9d2d95d6d52b41f59758a5469482621d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d14e47c24546435a907a918c15772ff2"
I20260925 01:34:08.815099 13420 raft_consensus.cc:3037] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:08.818135 13286 raft_consensus.cc:2454] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2d95d6d52b41f59758a5469482621d in term 1.
I20260925 01:34:08.818316 13420 raft_consensus.cc:2454] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d2d95d6d52b41f59758a5469482621d in term 1.
I20260925 01:34:08.818682 13086 leader_election.cc:302] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c06fb89f904c2db00a9f0ef6babbbd, 9d2d95d6d52b41f59758a5469482621d; no voters:
I20260925 01:34:08.819062 13492 raft_consensus.cc:2781] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:08.819653 13492 raft_consensus.cc:686] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 1 LEADER]: Becoming Leader. State: Replica: 9d2d95d6d52b41f59758a5469482621d, State: Running, Role: LEADER
I20260925 01:34:08.820307 13492 consensus_queue.cc:237] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } }
I20260925 01:34:08.825109 13007 catalog_manager.cc:5495] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d2d95d6d52b41f59758a5469482621d (127.27.160.1). New cstate: current_term: 1 leader_uuid: "9d2d95d6d52b41f59758a5469482621d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d2d95d6d52b41f59758a5469482621d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 43867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 } health_report { overall_health: UNKNOWN } } }
W20260925 01:34:08.853435 13198 tablet.cc:2369] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:34:08.883076 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.877223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.27.160.1:43867)
0925 01:34:08.877457 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.877462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.877483 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:34:08.877709 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.877713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.877796 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.877965 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:34:08.878359 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.879114 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.880599 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.881009 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.881165 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.881709 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:34:08.882424 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:34:08.882590 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.882699 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.882790 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:34:08.893347 13286 raft_consensus.cc:1261] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 1 FOLLOWER]: Refusing update from remote peer 9d2d95d6d52b41f59758a5469482621d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:08.893344 13420 raft_consensus.cc:1261] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 1 FOLLOWER]: Refusing update from remote peer 9d2d95d6d52b41f59758a5469482621d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:08.894470 13495 consensus_queue.cc:1035] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c06fb89f904c2db00a9f0ef6babbbd" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:08.894850 13492 consensus_queue.cc:1035] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d14e47c24546435a907a918c15772ff2" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 40641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:08.923300 13007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:08.923619 13007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:08.927384 13007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcba12fdc3e645c199bc165dda5d5007: Sending DeleteTablet for 3 replicas of tablet 229976d99b3f4904acb5bfecb7a007eb
I20260925 01:34:08.928725 13400 tablet_service.cc:1514] Processing DeleteTablet for tablet 229976d99b3f4904acb5bfecb7a007eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:08 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:52100
I20260925 01:34:08.928709 13132 tablet_service.cc:1514] Processing DeleteTablet for tablet 229976d99b3f4904acb5bfecb7a007eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:08 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:51372
I20260925 01:34:08.929709 13509 tablet_replica.cc:330] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: stopping tablet replica
I20260925 01:34:08.929761 13266 tablet_service.cc:1514] Processing DeleteTablet for tablet 229976d99b3f4904acb5bfecb7a007eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:08 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:56946
I20260925 01:34:08.930205 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 13069
I20260925 01:34:08.931126 13509 raft_consensus.cc:2229] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:34:08.931900 13509 raft_consensus.cc:2258] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:08.932679 13508 tablet_replica.cc:330] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: stopping tablet replica
I20260925 01:34:08.933059 13508 raft_consensus.cc:2229] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:08.933674 13508 raft_consensus.cc:2258] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:08.933717 13510 tablet_replica.cc:330] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: stopping tablet replica
I20260925 01:34:08.934224 13510 raft_consensus.cc:2229] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:08.934760 13510 raft_consensus.cc:2258] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:08.936092 13508 ts_tablet_manager.cc:1905] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:08.936951 13510 ts_tablet_manager.cc:1905] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:08.938915 13509 ts_tablet_manager.cc:1905] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:08.944509 13510 ts_tablet_manager.cc:1918] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:08.944823 13510 log.cc:1196] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/229976d99b3f4904acb5bfecb7a007eb
I20260925 01:34:08.945508 13510 ts_tablet_manager.cc:1939] T 229976d99b3f4904acb5bfecb7a007eb P 74c06fb89f904c2db00a9f0ef6babbbd: Deleting consensus metadata
I20260925 01:34:08.946878 13509 ts_tablet_manager.cc:1918] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:08.947075 13509 log.cc:1196] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/229976d99b3f4904acb5bfecb7a007eb
I20260925 01:34:08.947649 13509 ts_tablet_manager.cc:1939] T 229976d99b3f4904acb5bfecb7a007eb P 9d2d95d6d52b41f59758a5469482621d: Deleting consensus metadata
I20260925 01:34:08.948374 12993 catalog_manager.cc:4841] TS 74c06fb89f904c2db00a9f0ef6babbbd (127.27.160.2:43281): tablet 229976d99b3f4904acb5bfecb7a007eb (table test-table [id=baa02329291f483fbd157bbf618c3ef7]) successfully deleted
I20260925 01:34:08.949608 12992 catalog_manager.cc:4841] TS 9d2d95d6d52b41f59758a5469482621d (127.27.160.1:43867): tablet 229976d99b3f4904acb5bfecb7a007eb (table test-table [id=baa02329291f483fbd157bbf618c3ef7]) successfully deleted
I20260925 01:34:08.954643 13508 ts_tablet_manager.cc:1918] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:08.954830 13508 log.cc:1196] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/229976d99b3f4904acb5bfecb7a007eb
I20260925 01:34:08.955338 13508 ts_tablet_manager.cc:1939] T 229976d99b3f4904acb5bfecb7a007eb P d14e47c24546435a907a918c15772ff2: Deleting consensus metadata
I20260925 01:34:08.957024 12993 catalog_manager.cc:4841] TS d14e47c24546435a907a918c15772ff2 (127.27.160.3:40641): tablet 229976d99b3f4904acb5bfecb7a007eb (table test-table [id=baa02329291f483fbd157bbf618c3ef7]) successfully deleted
Sep 25 01:34:08 dist-test-slave-h6hf krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:34:08.969770 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:08.931649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.27.160.1:43867)
0925 01:34:08.931869 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:08.931877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:08.945871 (+ 13994us) server_negotiation.cc:407] Connection header received
0925 01:34:08.946106 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:08.946112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:08.946223 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:08.946326 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:08.948355 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.949689 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.951776 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:08.952558 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:08.952711 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:08.962160 (+ 9449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:08.962179 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:08.962196 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:08.962248 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:08.966140 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:08.967089 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:08.967108 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:08.967111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:08.967314 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:08.967809 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:08.967815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:08.967819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:08.968285 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:08.968661 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:08.969037 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 01:34:08.969381 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:34:08.970688 13192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:09.073237 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 13069
I20260925 01:34:09.092777 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 13203
Sep 25 01:34:09 dist-test-slave-h6hf krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:34:09.111260 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:09.093830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56948 (local address 127.27.160.2:43281)
0925 01:34:09.094049 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:09.094058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:09.094083 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:34:09.094157 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:09.094178 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:09.094263 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:09.094388 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:09.095881 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:09.096846 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:09.098545 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:09.099000 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:09.099156 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:09.104153 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:09.104183 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:09.104201 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:09.104246 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:09.107336 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:09.108270 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:09.108279 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:09.108281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:09.108462 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:09.108907 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:09.108914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:09.108917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:09.109811 (+ 894us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:09.110168 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:09.110527 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 01:34:09.110790 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:34:09.112216 13326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:09.207978 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 13203
I20260925 01:34:09.226887 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 13337
Sep 25 01:34:09 dist-test-slave-h6hf krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:34:09.244719 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:09.227897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.27.160.3:40641)
0925 01:34:09.228094 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:09.228099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:09.228119 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:34:09.228277 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:09.228281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:09.228358 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:09.228451 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:09.230152 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:09.230941 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:09.232495 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:09.232928 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:09.233033 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:09.238604 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:09.238624 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:09.238642 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:09.238685 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:09.241912 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:09.242723 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:09.242734 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:09.242738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:09.242892 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:09.243318 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:09.243322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:09.243325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:09.243744 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:09.243972 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:09.244274 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 01:34:09.244447 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:34:09.245468 13460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:09.336895 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 13337
I20260925 01:34:09.355844 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 12976
I20260925 01:34:09.357051 13038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:09.472162 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 12976
2026-09-25T01:34:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4779 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T01:34:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:09Z Disabled control of system clock
I20260925 01:34:09.564819 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:43599
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:43599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:34:09.918247 13519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:09.918568 13519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:09.918653 13519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:09.926285 13519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:09.926407 13519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:09.926445 13519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:09.926496 13519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:09.926539 13519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:09.936131 13519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:43599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:43599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:09.937604 13519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:09.939445 13519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:09.952111 13524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:09.952286 13525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:09.952239 13527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:09.953127 13519 server_base.cc:1034] running on GCE node
I20260925 01:34:09.954011 13519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:09.955534 13519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:09.956825 13519 hybrid_clock.cc:648] HybridClock initialized: now 1790300049956750 us; error 55 us; skew 500 ppm
I20260925 01:34:09.957374 13519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:09.960135 13519 webserver.cc:460] Webserver started at http://127.27.160.62:39411/ using document root <none> and password file <none>
I20260925 01:34:09.960969 13519 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:09.961185 13519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:09.961659 13519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:09.965087 13519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "76fff88a16ee4587b6e747c611fc85d1"
format_stamp: "Formatted at 2026-09-25 01:34:09 on dist-test-slave-h6hf"
server_key: "0ec067b5bdfa9441d097251472040da2"
server_key_iv: "82635e691b88b4fa46ae97ee8cbe750c"
server_key_version: "encryptionkey@0"
I20260925 01:34:09.966173 13519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "76fff88a16ee4587b6e747c611fc85d1"
format_stamp: "Formatted at 2026-09-25 01:34:09 on dist-test-slave-h6hf"
server_key: "0ec067b5bdfa9441d097251472040da2"
server_key_iv: "82635e691b88b4fa46ae97ee8cbe750c"
server_key_version: "encryptionkey@0"
I20260925 01:34:09.972311 13519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 01:34:09.976222 13533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:09.977936 13519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:09.978150 13519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "76fff88a16ee4587b6e747c611fc85d1"
format_stamp: "Formatted at 2026-09-25 01:34:09 on dist-test-slave-h6hf"
server_key: "0ec067b5bdfa9441d097251472040da2"
server_key_iv: "82635e691b88b4fa46ae97ee8cbe750c"
server_key_version: "encryptionkey@0"
I20260925 01:34:09.978413 13519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:09.996464 13519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:09.997442 13519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:09.997759 13519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:10.017339 13519 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:43599
I20260925 01:34:10.017365 13584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:43599 every 8 connection(s)
I20260925 01:34:10.018827 13519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:10.023029 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 13519
I20260925 01:34:10.023144 13585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:10.023176 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:10.023691 28288 external_mini_cluster.cc:1428] Setting key 24ea4d9f97d0be6bfabd0f3e582e2788
I20260925 01:34:10.041056 13585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1: Bootstrap starting.
I20260925 01:34:10.041198 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:10.025890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.27.160.62:43599)
0925 01:34:10.027227 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:10.027244 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:10.027309 (+ 65us) server_negotiation.cc:407] Connection header received
0925 01:34:10.029404 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:10.029451 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:10.030232 (+ 781us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:10.030781 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:10.031469 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:10.033788 (+ 2319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:10.035217 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:10.036433 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:10.036671 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:10.037941 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:10.037953 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:10.037974 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:10.038837 (+ 863us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:10.038959 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:10.038988 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:10.039122 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:10.039356 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 01:34:10.039741 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":440,"threads_started":1}
I20260925 01:34:10.044659 13585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:10.045759 13585 log.cc:824] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:10.048596 13585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1: No bootstrap required, opened a new log
I20260925 01:34:10.056910 13585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fff88a16ee4587b6e747c611fc85d1" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43599 } }
I20260925 01:34:10.057440 13585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:10.057534 13585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fff88a16ee4587b6e747c611fc85d1, State: Initialized, Role: FOLLOWER
I20260925 01:34:10.058178 13585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fff88a16ee4587b6e747c611fc85d1" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43599 } }
I20260925 01:34:10.058401 13585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:10.058508 13585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:10.058657 13585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:10.061080 13585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fff88a16ee4587b6e747c611fc85d1" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43599 } }
I20260925 01:34:10.061705 13585 leader_election.cc:302] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76fff88a16ee4587b6e747c611fc85d1; no voters:
I20260925 01:34:10.062508 13585 leader_election.cc:288] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:10.062669 13590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:10.063493 13590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [term 1 LEADER]: Becoming Leader. State: Replica: 76fff88a16ee4587b6e747c611fc85d1, State: Running, Role: LEADER
I20260925 01:34:10.064013 13590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fff88a16ee4587b6e747c611fc85d1" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43599 } }
I20260925 01:34:10.064862 13585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:10.067211 13591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76fff88a16ee4587b6e747c611fc85d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fff88a16ee4587b6e747c611fc85d1" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43599 } } }
I20260925 01:34:10.067147 13592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76fff88a16ee4587b6e747c611fc85d1. Latest consensus state: current_term: 1 leader_uuid: "76fff88a16ee4587b6e747c611fc85d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76fff88a16ee4587b6e747c611fc85d1" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43599 } } }
I20260925 01:34:10.067728 13591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:10.067749 13592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:10.069456 13598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:10.078549 13598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:10.089473 13598 catalog_manager.cc:1347] Generated new cluster ID: 1821ac31a01c4b1887b56fb83dc6f2bb
I20260925 01:34:10.089635 13598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:10.113864 13598 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:10.115202 13598 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:10.132164 13598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1: Generated new TSK 0
I20260925 01:34:10.133203 13598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:10.142339 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43599
--builtin_ntp_servers=127.27.160.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:34:10.498698 13609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:10.499011 13609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:10.499167 13609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:10.506791 13609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:10.506908 13609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:10.507081 13609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:34:10.516840 13609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:10.518261 13609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:10.520238 13609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:10.534652 13617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:10.534809 13614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:10.534759 13615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:10.535504 13609 server_base.cc:1034] running on GCE node
I20260925 01:34:10.536180 13609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:10.537284 13609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:10.538522 13609 hybrid_clock.cc:648] HybridClock initialized: now 1790300050538451 us; error 60 us; skew 500 ppm
I20260925 01:34:10.538977 13609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:10.542068 13609 webserver.cc:460] Webserver started at http://127.27.160.1:46215/ using document root <none> and password file <none>
I20260925 01:34:10.542886 13609 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:10.543012 13609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:10.543373 13609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:10.546496 13609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "2a92f568604d4e5ab9534cea09bdff65"
format_stamp: "Formatted at 2026-09-25 01:34:10 on dist-test-slave-h6hf"
server_key: "c4379837fd950e2b864bfe0027ff77a0"
server_key_iv: "51d2fdc349d58d14426eed07b7c961e2"
server_key_version: "encryptionkey@0"
I20260925 01:34:10.547354 13609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "2a92f568604d4e5ab9534cea09bdff65"
format_stamp: "Formatted at 2026-09-25 01:34:10 on dist-test-slave-h6hf"
server_key: "c4379837fd950e2b864bfe0027ff77a0"
server_key_iv: "51d2fdc349d58d14426eed07b7c961e2"
server_key_version: "encryptionkey@0"
I20260925 01:34:10.552979 13609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:34:10.556747 13623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:10.558527 13609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:10.558741 13609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "2a92f568604d4e5ab9534cea09bdff65"
format_stamp: "Formatted at 2026-09-25 01:34:10 on dist-test-slave-h6hf"
server_key: "c4379837fd950e2b864bfe0027ff77a0"
server_key_iv: "51d2fdc349d58d14426eed07b7c961e2"
server_key_version: "encryptionkey@0"
I20260925 01:34:10.559000 13609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:10.575862 13609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:10.576717 13609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:10.577035 13609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:10.578141 13609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:10.579671 13609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:10.579795 13609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:10.579936 13609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:10.580026 13609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:10.615686 13609 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:45179
I20260925 01:34:10.615792 13735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:45179 every 8 connection(s)
I20260925 01:34:10.617152 13609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:34:10.623019 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 13609
I20260925 01:34:10.623217 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:34:10.623675 28288 external_mini_cluster.cc:1428] Setting key ee1db21dd7bf2401ac61d42a0dd55d8a
I20260925 01:34:10.626930 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43599
--builtin_ntp_servers=127.27.160.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:34:10.632489 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:10.619960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:44437 (local address 127.27.160.62:43599)
0925 01:34:10.620523 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:10.620529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:10.621709 (+ 1180us) server_negotiation.cc:407] Connection header received
0925 01:34:10.623606 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:10.623612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:10.623714 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:10.623851 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:10.625324 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:10.626235 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:10.628850 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:10.629421 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:10.629540 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:10.631428 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:10.631434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:10.631438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:10.631461 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:10.631504 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:10.631517 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:10.631738 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:10.631909 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 01:34:10.632052 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":282,"threads_started":1}
I20260925 01:34:10.633972 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:10.620489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43599 (local address 127.27.160.1:44437)
0925 01:34:10.621489 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:10.621526 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:10.623303 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:10.624148 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:10.624161 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:10.624337 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:10.625082 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:10.625102 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:10.626404 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:10.626409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:10.628655 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:10.628674 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:10.629683 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:10.629693 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:10.630049 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:10.630992 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:10.630998 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:10.631060 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:10.631178 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:10.631629 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:10.631643 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:10.631654 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:34:10.631870 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 01:34:10.632384 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":269,"threads_started":1}
I20260925 01:34:10.635982 13736 heartbeater.cc:347] Connected to a master server at 127.27.160.62:43599
I20260925 01:34:10.636409 13736 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:10.637571 13736 heartbeater.cc:511] Master 127.27.160.62:43599 requested a full tablet report, sending...
I20260925 01:34:10.640235 13550 ts_manager.cc:194] Registered new tserver with Master: 2a92f568604d4e5ab9534cea09bdff65 (127.27.160.1:45179)
I20260925 01:34:10.642493 13550 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.1:44437
W20260925 01:34:10.967639 13740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:10.967918 13740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:10.968003 13740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:10.975579 13740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:10.975688 13740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:10.975798 13740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:34:10.985239 13740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:10.986598 13740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:10.988368 13740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:11.001789 13745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:11.001822 13748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:11.001874 13746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:11.003165 13740 server_base.cc:1034] running on GCE node
I20260925 01:34:11.003975 13740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:11.005306 13740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:11.006584 13740 hybrid_clock.cc:648] HybridClock initialized: now 1790300051006519 us; error 55 us; skew 500 ppm
I20260925 01:34:11.007028 13740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:11.009701 13740 webserver.cc:460] Webserver started at http://127.27.160.2:34793/ using document root <none> and password file <none>
I20260925 01:34:11.010479 13740 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:11.010618 13740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:11.010954 13740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:11.014040 13740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "9a8779529d2c4bab8a1a1c149583a3cb"
format_stamp: "Formatted at 2026-09-25 01:34:11 on dist-test-slave-h6hf"
server_key: "bea86edeeb7cd12e3461349de34b7dfb"
server_key_iv: "0bd21326ca5902f98bee3a771dd37d1e"
server_key_version: "encryptionkey@0"
I20260925 01:34:11.014876 13740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "9a8779529d2c4bab8a1a1c149583a3cb"
format_stamp: "Formatted at 2026-09-25 01:34:11 on dist-test-slave-h6hf"
server_key: "bea86edeeb7cd12e3461349de34b7dfb"
server_key_iv: "0bd21326ca5902f98bee3a771dd37d1e"
server_key_version: "encryptionkey@0"
I20260925 01:34:11.020661 13740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 01:34:11.024569 13754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:11.026240 13740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:11.026425 13740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "9a8779529d2c4bab8a1a1c149583a3cb"
format_stamp: "Formatted at 2026-09-25 01:34:11 on dist-test-slave-h6hf"
server_key: "bea86edeeb7cd12e3461349de34b7dfb"
server_key_iv: "0bd21326ca5902f98bee3a771dd37d1e"
server_key_version: "encryptionkey@0"
I20260925 01:34:11.026674 13740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:11.050920 13740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:11.051860 13740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:11.052177 13740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:11.053262 13740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:11.054808 13740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:11.054893 13740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:11.055047 13740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:11.055142 13740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:11.090940 13740 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:36707
I20260925 01:34:11.091037 13866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:36707 every 8 connection(s)
I20260925 01:34:11.092319 13740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:34:11.098912 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 13740
I20260925 01:34:11.099130 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:34:11.099778 28288 external_mini_cluster.cc:1428] Setting key 948244f4c156fb041e4b1eb7c96157d1
I20260925 01:34:11.103381 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43599
--builtin_ntp_servers=127.27.160.20:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:34:11.108234 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.095122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:48167 (local address 127.27.160.62:43599)
0925 01:34:11.095360 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.095366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.096837 (+ 1471us) server_negotiation.cc:407] Connection header received
0925 01:34:11.099109 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.099115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.099235 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.099340 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:11.100901 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.102039 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.104795 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.105254 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.105391 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.107196 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:11.107201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:11.107204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:11.107223 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:11.107256 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:11.107269 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:11.107340 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.107699 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.107824 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 01:34:11.109766 13869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.095631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43599 (local address 127.27.160.2:48167)
0925 01:34:11.096623 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:11.096670 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:11.098693 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:11.099643 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:11.099656 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:11.099818 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:11.100698 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.100725 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.102218 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.102223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.104597 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.104618 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.105546 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.105557 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.105952 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.106780 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:11.106787 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:11.106856 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:11.106983 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:11.107372 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:11.107386 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:11.107398 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 01:34:11.107612 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 01:34:11.108112 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":278,"threads_started":1}
I20260925 01:34:11.111338 13867 heartbeater.cc:347] Connected to a master server at 127.27.160.62:43599
I20260925 01:34:11.111712 13867 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:11.112749 13867 heartbeater.cc:511] Master 127.27.160.62:43599 requested a full tablet report, sending...
I20260925 01:34:11.114943 13550 ts_manager.cc:194] Registered new tserver with Master: 9a8779529d2c4bab8a1a1c149583a3cb (127.27.160.2:36707)
I20260925 01:34:11.116102 13550 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.2:48167
W20260925 01:34:11.438488 13870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:11.438855 13870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:11.439028 13870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:11.446344 13870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:11.446514 13870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:11.446712 13870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:34:11.456070 13870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:43599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:11.457481 13870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:11.459390 13870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:11.473408 13875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:11.473361 13878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:11.473378 13876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:11.474773 13870 server_base.cc:1034] running on GCE node
I20260925 01:34:11.475432 13870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:11.476464 13870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:11.477700 13870 hybrid_clock.cc:648] HybridClock initialized: now 1790300051477659 us; error 42 us; skew 500 ppm
I20260925 01:34:11.478037 13870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:11.480458 13870 webserver.cc:460] Webserver started at http://127.27.160.3:41903/ using document root <none> and password file <none>
I20260925 01:34:11.481244 13870 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:11.481360 13870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:11.481700 13870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:11.484704 13870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "c4b2e03b514b4cf499662a0a7ad3e140"
format_stamp: "Formatted at 2026-09-25 01:34:11 on dist-test-slave-h6hf"
server_key: "4f31c499d522b1c3dc159cadc4776a3f"
server_key_iv: "ed71505d4882a829d69ccff3d81046e5"
server_key_version: "encryptionkey@0"
I20260925 01:34:11.485548 13870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "c4b2e03b514b4cf499662a0a7ad3e140"
format_stamp: "Formatted at 2026-09-25 01:34:11 on dist-test-slave-h6hf"
server_key: "4f31c499d522b1c3dc159cadc4776a3f"
server_key_iv: "ed71505d4882a829d69ccff3d81046e5"
server_key_version: "encryptionkey@0"
I20260925 01:34:11.490985 13870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:34:11.495097 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:11.496589 13870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:34:11.496786 13870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "c4b2e03b514b4cf499662a0a7ad3e140"
format_stamp: "Formatted at 2026-09-25 01:34:11 on dist-test-slave-h6hf"
server_key: "4f31c499d522b1c3dc159cadc4776a3f"
server_key_iv: "ed71505d4882a829d69ccff3d81046e5"
server_key_version: "encryptionkey@0"
I20260925 01:34:11.497036 13870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:11.518389 13870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:11.519305 13870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:11.519634 13870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:11.520718 13870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:11.522469 13870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:11.522593 13870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:11.522734 13870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:11.522825 13870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:11.567916 13870 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:36037
I20260925 01:34:11.568017 13996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:36037 every 8 connection(s)
I20260925 01:34:11.569566 13870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:34:11.577585 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 13870
I20260925 01:34:11.577849 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:34:11.578358 28288 external_mini_cluster.cc:1428] Setting key 651beeb3ff089be9f63fb687ee5d4015
I20260925 01:34:11.584671 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.572448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:42591 (local address 127.27.160.62:43599)
0925 01:34:11.572675 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.572680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.574420 (+ 1740us) server_negotiation.cc:407] Connection header received
0925 01:34:11.576394 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.576400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.576485 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.576585 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:11.578028 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.578994 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.581440 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.582042 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.582232 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.583880 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:11.583885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:11.583889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:11.583909 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:11.583945 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:11.583959 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:11.584038 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.584335 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.584420 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:34:11.586163 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.573078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43599 (local address 127.27.160.3:42591)
0925 01:34:11.574245 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:11.574284 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:11.576056 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:11.576882 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:11.576892 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:11.576993 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:11.577795 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.577827 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.579256 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.579263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.581255 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.581290 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.582341 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.582350 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.582689 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.583498 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:11.583501 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:11.583547 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:11.583644 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:11.584075 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:11.584084 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:11.584091 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:34:11.584255 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 01:34:11.584598 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":372,"threads_started":1}
I20260925 01:34:11.587678 13997 heartbeater.cc:347] Connected to a master server at 127.27.160.62:43599
I20260925 01:34:11.588011 13997 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:11.588812 13997 heartbeater.cc:511] Master 127.27.160.62:43599 requested a full tablet report, sending...
I20260925 01:34:11.590399 13550 ts_manager.cc:194] Registered new tserver with Master: c4b2e03b514b4cf499662a0a7ad3e140 (127.27.160.3:36037)
I20260925 01:34:11.591384 13550 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.3:42591
I20260925 01:34:11.594457 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:34:11.605273 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.598895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.27.160.62:43599)
0925 01:34:11.599127 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.599155 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.599578 (+ 423us) server_negotiation.cc:407] Connection header received
0925 01:34:11.599834 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.599840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.599926 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.600030 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:11.600551 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.601342 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.602731 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.603167 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.603305 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.604483 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:11.604493 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:11.604496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:11.604515 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:11.604548 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:11.604560 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:11.604660 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.604805 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.604900 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:34:11.609392 13550 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:34:11.616243 13550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:34:11.640120 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.629593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:36037 (local address 127.0.0.1:58630)
0925 01:34:11.630362 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:11.630397 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:11.630566 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:11.632059 (+ 1493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:11.632070 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:11.632111 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:11.632631 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.632648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.634876 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.634883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.637381 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.637395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.638836 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.638843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.639037 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.639090 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 01:34:11.639498 (+ 408us) client_negotiation.cc:240] Negotiation successful
0925 01:34:11.639728 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":24,"thread_start_us":343,"threads_started":1}
I20260925 01:34:11.641019 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.629916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.27.160.3:36037)
0925 01:34:11.631259 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.631273 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.631317 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:34:11.631438 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.631450 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.631768 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.631998 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:11.632902 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.634700 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.637598 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.638609 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.638762 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.639062 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.640375 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.640577 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":953,"threads_started":1}
I20260925 01:34:11.641723 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.630942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:45179 (local address 127.0.0.1:50878)
0925 01:34:11.632781 (+ 1839us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:11.632799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:11.632918 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:11.633896 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:11.633901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:11.633926 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:11.634456 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.634470 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.636443 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.636448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.638758 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.638773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.641008 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.641017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.641249 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.641286 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:34:11.641363 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 01:34:11.641463 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1681,"mutex_wait_us":2,"thread_start_us":1521,"threads_started":1}
I20260925 01:34:11.642151 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.631114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.27.160.1:45179)
0925 01:34:11.631671 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.631682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.633038 (+ 1356us) server_negotiation.cc:407] Connection header received
0925 01:34:11.633176 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.633185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.633425 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.633619 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:11.634659 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.636264 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.638987 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.640800 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.640942 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.641516 (+ 574us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.641602 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.641770 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":357,"threads_started":1}
I20260925 01:34:11.643719 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.630073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:36707 (local address 127.0.0.1:39722)
0925 01:34:11.634550 (+ 4477us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:11.634569 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:11.634690 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:11.635218 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:11.635223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:11.635247 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:11.636141 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.636156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.637743 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.637748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.639837 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.639853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.642871 (+ 3018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.642877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.643053 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.643125 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 01:34:11.643222 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 01:34:11.643364 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4329,"mutex_wait_us":1530,"thread_start_us":2875,"threads_started":1}
I20260925 01:34:11.643924 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.630543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.27.160.2:36707)
0925 01:34:11.634023 (+ 3480us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.634033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.634678 (+ 645us) server_negotiation.cc:407] Connection header received
0925 01:34:11.634830 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.634836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.634993 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.635231 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:11.636333 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.637594 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.640083 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.642660 (+ 2577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.642830 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.643086 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.643286 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.643496 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3310,"thread_start_us":3197,"threads_started":1}
I20260925 01:34:11.645730 13932 tablet_service.cc:1467] Processing CreateTablet for tablet 858e41988d904f60acb9db88feb9b698 (DEFAULT_TABLE table=test-table [id=6f9bb09ac6a74bfd87283e06c90fc6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:11.646868 13736 heartbeater.cc:503] Master 127.27.160.62:43599 was elected leader, sending a full tablet report...
I20260925 01:34:11.647289 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858e41988d904f60acb9db88feb9b698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:11.647701 13802 tablet_service.cc:1467] Processing CreateTablet for tablet 858e41988d904f60acb9db88feb9b698 (DEFAULT_TABLE table=test-table [id=6f9bb09ac6a74bfd87283e06c90fc6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:11.649076 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858e41988d904f60acb9db88feb9b698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:11.652146 13671 tablet_service.cc:1467] Processing CreateTablet for tablet 858e41988d904f60acb9db88feb9b698 (DEFAULT_TABLE table=test-table [id=6f9bb09ac6a74bfd87283e06c90fc6d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:11.653757 13671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 858e41988d904f60acb9db88feb9b698. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:11.666262 14015 tablet_bootstrap.cc:492] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: Bootstrap starting.
I20260925 01:34:11.670051 14016 tablet_bootstrap.cc:492] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: Bootstrap starting.
I20260925 01:34:11.671082 14015 tablet_bootstrap.cc:654] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:11.672202 14015 log.cc:824] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:11.673074 14017 tablet_bootstrap.cc:492] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: Bootstrap starting.
I20260925 01:34:11.674362 14016 tablet_bootstrap.cc:654] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:11.675657 14016 log.cc:824] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:11.676564 14015 tablet_bootstrap.cc:492] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: No bootstrap required, opened a new log
I20260925 01:34:11.676884 14015 ts_tablet_manager.cc:1397] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260925 01:34:11.678951 14017 tablet_bootstrap.cc:654] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:11.679237 14016 tablet_bootstrap.cc:492] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: No bootstrap required, opened a new log
I20260925 01:34:11.679558 14016 ts_tablet_manager.cc:1397] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:34:11.680358 14017 log.cc:824] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:11.683879 14017 tablet_bootstrap.cc:492] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: No bootstrap required, opened a new log
I20260925 01:34:11.684226 14017 ts_tablet_manager.cc:1397] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 01:34:11.685503 14015 raft_consensus.cc:348] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.685979 14015 raft_consensus.cc:374] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:11.686085 14015 raft_consensus.cc:729] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8779529d2c4bab8a1a1c149583a3cb, State: Initialized, Role: FOLLOWER
I20260925 01:34:11.686766 14015 consensus_queue.cc:260] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.687983 13867 heartbeater.cc:503] Master 127.27.160.62:43599 was elected leader, sending a full tablet report...
I20260925 01:34:11.688474 14015 ts_tablet_manager.cc:1428] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 01:34:11.688943 14021 raft_consensus.cc:482] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:34:11.689270 14021 raft_consensus.cc:504] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.692075 14016 raft_consensus.cc:348] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.692624 14016 raft_consensus.cc:374] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:11.692746 14016 raft_consensus.cc:729] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a92f568604d4e5ab9534cea09bdff65, State: Initialized, Role: FOLLOWER
I20260925 01:34:11.693682 14016 consensus_queue.cc:260] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.695859 14017 raft_consensus.cc:348] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.696337 14017 raft_consensus.cc:374] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:11.696458 14017 raft_consensus.cc:729] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b2e03b514b4cf499662a0a7ad3e140, State: Initialized, Role: FOLLOWER
I20260925 01:34:11.697144 14017 consensus_queue.cc:260] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.698345 13997 heartbeater.cc:503] Master 127.27.160.62:43599 was elected leader, sending a full tablet report...
I20260925 01:34:11.699090 14017 ts_tablet_manager.cc:1428] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260925 01:34:11.699049 14021 leader_election.cc:288] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4b2e03b514b4cf499662a0a7ad3e140 (127.27.160.3:36037), 2a92f568604d4e5ab9534cea09bdff65 (127.27.160.1:45179)
I20260925 01:34:11.704087 14016 ts_tablet_manager.cc:1428] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: Time spent starting tablet: real 0.024s user 0.018s sys 0.005s
I20260925 01:34:11.707406 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.691968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:54875 (local address 127.27.160.3:36037)
0925 01:34:11.692168 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.692174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.700521 (+ 8347us) server_negotiation.cc:407] Connection header received
0925 01:34:11.700748 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.700757 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.700869 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.700993 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:11.701894 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.702990 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.704994 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.706226 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.706406 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.706484 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.706988 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.707141 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:34:11.707477 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.691521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:36037 (local address 127.27.160.2:54875)
0925 01:34:11.700391 (+ 8870us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:11.700419 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:11.700586 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:11.701220 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:11.701226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:11.701258 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:11.701691 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.701708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.703212 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.703217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.704810 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.704823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.706458 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.706470 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.706773 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.706830 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 01:34:11.706958 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 01:34:11.707115 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8696,"thread_start_us":8747,"threads_started":1}
I20260925 01:34:11.708792 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e41988d904f60acb9db88feb9b698" candidate_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" is_pre_election: true
I20260925 01:34:11.709455 13952 raft_consensus.cc:2454] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a8779529d2c4bab8a1a1c149583a3cb in term 0.
I20260925 01:34:11.710180 13755 leader_election.cc:302] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a8779529d2c4bab8a1a1c149583a3cb, c4b2e03b514b4cf499662a0a7ad3e140; no voters:
I20260925 01:34:11.710623 14021 raft_consensus.cc:2781] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:34:11.710804 14021 raft_consensus.cc:482] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:34:11.710959 14021 raft_consensus.cc:3037] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:11.711412 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.702564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:45179 (local address 127.27.160.2:48237)
0925 01:34:11.705446 (+ 2882us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:11.705471 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:11.705601 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:11.706004 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:11.706011 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:11.706042 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:11.706429 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.706443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.707474 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.707479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.709216 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:11.709231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.710693 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.710700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:11.710881 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.710905 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:34:11.711020 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 01:34:11.711164 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2756,"thread_start_us":2579,"threads_started":1}
I20260925 01:34:11.711524 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.702204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:48237 (local address 127.27.160.1:45179)
0925 01:34:11.702416 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.702422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.705583 (+ 3161us) server_negotiation.cc:407] Connection header received
0925 01:34:11.705759 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.705765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.705856 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.705981 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:11.706578 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.707321 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.709442 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.710482 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.710655 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.710731 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.711063 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.711227 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:34:11.712471 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e41988d904f60acb9db88feb9b698" candidate_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a92f568604d4e5ab9534cea09bdff65" is_pre_election: true
I20260925 01:34:11.713083 13691 raft_consensus.cc:2454] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a8779529d2c4bab8a1a1c149583a3cb in term 0.
I20260925 01:34:11.714751 14021 raft_consensus.cc:504] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.715536 14021 leader_election.cc:288] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [CANDIDATE]: Term 1 election: Requested vote from peers c4b2e03b514b4cf499662a0a7ad3e140 (127.27.160.3:36037), 2a92f568604d4e5ab9534cea09bdff65 (127.27.160.1:45179)
I20260925 01:34:11.716064 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e41988d904f60acb9db88feb9b698" candidate_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4b2e03b514b4cf499662a0a7ad3e140"
I20260925 01:34:11.716182 13691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "858e41988d904f60acb9db88feb9b698" candidate_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a92f568604d4e5ab9534cea09bdff65"
I20260925 01:34:11.716329 13952 raft_consensus.cc:3037] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:11.716395 13691 raft_consensus.cc:3037] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:11.718822 13952 raft_consensus.cc:2454] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a8779529d2c4bab8a1a1c149583a3cb in term 1.
I20260925 01:34:11.718950 13691 raft_consensus.cc:2454] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a8779529d2c4bab8a1a1c149583a3cb in term 1.
I20260925 01:34:11.719324 13755 leader_election.cc:302] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a8779529d2c4bab8a1a1c149583a3cb, c4b2e03b514b4cf499662a0a7ad3e140; no voters:
I20260925 01:34:11.719695 14021 raft_consensus.cc:2781] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:11.720265 14021 raft_consensus.cc:686] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 1 LEADER]: Becoming Leader. State: Replica: 9a8779529d2c4bab8a1a1c149583a3cb, State: Running, Role: LEADER
I20260925 01:34:11.720898 14021 consensus_queue.cc:237] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } }
I20260925 01:34:11.725806 13550 catalog_manager.cc:5495] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a8779529d2c4bab8a1a1c149583a3cb (127.27.160.2). New cstate: current_term: 1 leader_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a8779529d2c4bab8a1a1c149583a3cb" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 36707 } health_report { overall_health: HEALTHY } } }
I20260925 01:34:11.749468 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.742788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.27.160.2:36707)
0925 01:34:11.742982 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.742987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.743005 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:34:11.743099 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.743102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.743242 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.743392 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:34:11.743801 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.744582 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.746682 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.747337 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.747492 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.748108 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:34:11.748668 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:34:11.748849 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.748950 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.749043 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:34:11.758962 13952 raft_consensus.cc:1261] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 1 FOLLOWER]: Refusing update from remote peer 9a8779529d2c4bab8a1a1c149583a3cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:11.759001 13691 raft_consensus.cc:1261] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 1 FOLLOWER]: Refusing update from remote peer 9a8779529d2c4bab8a1a1c149583a3cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:11.760129 14026 consensus_queue.cc:1035] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4b2e03b514b4cf499662a0a7ad3e140" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 36037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:11.760514 14021 consensus_queue.cc:1035] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a92f568604d4e5ab9534cea09bdff65" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:11.777416 14031 mvcc.cc:205] Tried to move back new op lower bound from 7333069011992522752 to 7333069011853537280. Current Snapshot: MvccSnapshot[applied={T|T < 7333069011992522752}]
I20260925 01:34:11.792080 13549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:11.792433 13549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:11.797140 13549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76fff88a16ee4587b6e747c611fc85d1: Sending DeleteTablet for 3 replicas of tablet 858e41988d904f60acb9db88feb9b698
I20260925 01:34:11.798440 13932 tablet_service.cc:1514] Processing DeleteTablet for tablet 858e41988d904f60acb9db88feb9b698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:11 UTC) from {username='slave'} at 127.0.0.1:58630
I20260925 01:34:11.798570 13671 tablet_service.cc:1514] Processing DeleteTablet for tablet 858e41988d904f60acb9db88feb9b698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:11 UTC) from {username='slave'} at 127.0.0.1:50878
I20260925 01:34:11.798837 13802 tablet_service.cc:1514] Processing DeleteTablet for tablet 858e41988d904f60acb9db88feb9b698 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:11 UTC) from {username='slave'} at 127.0.0.1:39722
I20260925 01:34:11.801465 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 13609
I20260925 01:34:11.810649 14041 tablet_replica.cc:330] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: stopping tablet replica
I20260925 01:34:11.811589 14041 raft_consensus.cc:2229] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:34:11.812777 14041 raft_consensus.cc:2258] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:11.813854 14040 tablet_replica.cc:330] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: stopping tablet replica
I20260925 01:34:11.814288 14040 raft_consensus.cc:2229] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:11.814793 14040 raft_consensus.cc:2258] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:11.817090 14040 ts_tablet_manager.cc:1905] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:11.817098 14041 ts_tablet_manager.cc:1905] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:11.825351 14041 ts_tablet_manager.cc:1918] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:11.825351 14040 ts_tablet_manager.cc:1918] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:11.825564 14041 log.cc:1196] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/858e41988d904f60acb9db88feb9b698
I20260925 01:34:11.825773 14040 log.cc:1196] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/858e41988d904f60acb9db88feb9b698
I20260925 01:34:11.826354 14041 ts_tablet_manager.cc:1939] T 858e41988d904f60acb9db88feb9b698 P 9a8779529d2c4bab8a1a1c149583a3cb: Deleting consensus metadata
I20260925 01:34:11.826534 14040 ts_tablet_manager.cc:1939] T 858e41988d904f60acb9db88feb9b698 P 2a92f568604d4e5ab9534cea09bdff65: Deleting consensus metadata
I20260925 01:34:11.828666 13534 catalog_manager.cc:4841] TS 2a92f568604d4e5ab9534cea09bdff65 (127.27.160.1:45179): tablet 858e41988d904f60acb9db88feb9b698 (table test-table [id=6f9bb09ac6a74bfd87283e06c90fc6d9]) successfully deleted
I20260925 01:34:11.829265 13537 catalog_manager.cc:4841] TS 9a8779529d2c4bab8a1a1c149583a3cb (127.27.160.2:36707): tablet 858e41988d904f60acb9db88feb9b698 (table test-table [id=6f9bb09ac6a74bfd87283e06c90fc6d9]) successfully deleted
I20260925 01:34:11.829730 14039 tablet_replica.cc:330] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: stopping tablet replica
I20260925 01:34:11.830340 14039 raft_consensus.cc:2229] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:11.830964 14039 raft_consensus.cc:2258] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:11.832402 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.802997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.27.160.1:45179)
0925 01:34:11.803210 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.803217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.822520 (+ 19303us) server_negotiation.cc:407] Connection header received
0925 01:34:11.823101 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.823108 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.823272 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.823360 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:11.824171 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.825474 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.828237 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.828923 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.829035 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.830699 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:11.830709 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:11.830729 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:11.831541 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:11.831620 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:11.831646 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:11.831764 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.831988 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.832117 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:34:11.833233 13731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:11.834321 14039 ts_tablet_manager.cc:1905] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:11.843212 14039 ts_tablet_manager.cc:1918] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:11.843463 14039 log.cc:1196] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/858e41988d904f60acb9db88feb9b698
I20260925 01:34:11.844139 14039 ts_tablet_manager.cc:1939] T 858e41988d904f60acb9db88feb9b698 P c4b2e03b514b4cf499662a0a7ad3e140: Deleting consensus metadata
I20260925 01:34:11.846642 13534 catalog_manager.cc:4841] TS c4b2e03b514b4cf499662a0a7ad3e140 (127.27.160.3:36037): tablet 858e41988d904f60acb9db88feb9b698 (table test-table [id=6f9bb09ac6a74bfd87283e06c90fc6d9]) successfully deleted
I20260925 01:34:11.931955 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 13609
I20260925 01:34:11.951627 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 13740
I20260925 01:34:11.960498 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:11.952480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.27.160.2:36707)
0925 01:34:11.952669 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:11.952674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:11.952845 (+ 171us) server_negotiation.cc:407] Connection header received
0925 01:34:11.953047 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:11.953051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:11.953327 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:11.953423 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:11.953935 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.954707 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.956899 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:11.957479 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:11.957584 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:11.958888 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:11.958897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:11.958915 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:11.959532 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:11.959584 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:11.959605 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:11.959699 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:11.959955 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 01:34:11.960056 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:34:11.961359 13862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:12.079291 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 13740
I20260925 01:34:12.097821 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 13870
I20260925 01:34:12.105418 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:12.098770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.27.160.3:36037)
0925 01:34:12.098956 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:12.098961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:12.098979 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:34:12.099050 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:12.099057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:12.099147 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:12.099251 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:12.099727 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:12.100566 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:12.102232 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:12.102683 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:12.102786 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:12.104009 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:12.104019 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:12.104034 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:12.104637 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:12.104688 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:12.104709 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:12.104800 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:12.104964 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 01:34:12.105055 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:34:12.106117 13992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:12.197739 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 13870
I20260925 01:34:12.215098 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 13519
I20260925 01:34:12.216362 13580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:12.328151 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 13519
2026-09-25T01:34:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2850 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:12 dist-test-slave-h6hf krb5kdc[14049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:12 dist-test-slave-h6hf krb5kdc[14049](info): set up 2 sockets
Sep 25 01:34:12 dist-test-slave-h6hf krb5kdc[14049](info): commencing operation
krb5kdc: starting...
W20260925 01:34:13.442591 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:13 dist-test-slave-h6hf krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:34:13.732882 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:35907
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:14.113055 14062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:14.113417 14062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:14.113565 14062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:14.121881 14062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:14.122056 14062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:14.122175 14062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:14.122293 14062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:14.122375 14062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:14.133030 14062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:35907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:14.134541 14062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:14.135944 14062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:14.148970 14067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:14.149046 14068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:14.148993 14070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:14.150023 14062 server_base.cc:1034] running on GCE node
I20260925 01:34:14.150964 14062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:14.152573 14062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:14.153854 14062 hybrid_clock.cc:648] HybridClock initialized: now 1790300054153759 us; error 69 us; skew 500 ppm
I20260925 01:34:14.154285 14062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:14 dist-test-slave-h6hf krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300054, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:14.165007 14062 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:34:14.166957 14062 webserver.cc:460] Webserver started at http://127.27.160.62:34141/ using document root <none> and password file <none>
I20260925 01:34:14.167757 14062 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:14.167871 14062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:14.168223 14062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:14.171506 14062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "8175f12557bb423a804b9731582c10e4"
format_stamp: "Formatted at 2026-09-25 01:34:14 on dist-test-slave-h6hf"
server_key: "d9999e93f52b9f3ab4b162914e97a9c9"
server_key_iv: "8aa1fea73aea38c40a02793e92727035"
server_key_version: "encryptionkey@0"
I20260925 01:34:14.172323 14062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "8175f12557bb423a804b9731582c10e4"
format_stamp: "Formatted at 2026-09-25 01:34:14 on dist-test-slave-h6hf"
server_key: "d9999e93f52b9f3ab4b162914e97a9c9"
server_key_iv: "8aa1fea73aea38c40a02793e92727035"
server_key_version: "encryptionkey@0"
I20260925 01:34:14.178637 14062 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 01:34:14.182687 14077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:14.184203 14062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:14.184499 14062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "8175f12557bb423a804b9731582c10e4"
format_stamp: "Formatted at 2026-09-25 01:34:14 on dist-test-slave-h6hf"
server_key: "d9999e93f52b9f3ab4b162914e97a9c9"
server_key_iv: "8aa1fea73aea38c40a02793e92727035"
server_key_version: "encryptionkey@0"
I20260925 01:34:14.184731 14062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:14.217106 14062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:14.222123 14062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:14.222440 14062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:14.242465 14062 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:35907
I20260925 01:34:14.242503 14128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:35907 every 8 connection(s)
I20260925 01:34:14.243947 14062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:14.248135 14129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:14.251798 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 14062
I20260925 01:34:14.252024 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:14.252656 28288 external_mini_cluster.cc:1428] Setting key f3b3b4b9df01b5109e9b48bb64bd83e3
I20260925 01:34:14.262758 14129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4: Bootstrap starting.
I20260925 01:34:14.267308 14129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:14.268868 14129 log.cc:824] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:34:14 dist-test-slave-h6hf krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:14.272666 14129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4: No bootstrap required, opened a new log
I20260925 01:34:14.280241 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:14.254967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.27.160.62:35907)
0925 01:34:14.255824 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:14.255836 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:14.255876 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:34:14.257737 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:14.257763 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:14.258193 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:14.258558 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:14.260698 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:14.262168 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:14.264282 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:14.265191 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:14.265366 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:14.271293 (+ 5927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:14.271301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:14.271317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:14.271355 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:14.275370 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:14.276406 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:14.276420 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:14.276427 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:14.276616 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:14.277206 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:14.277210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:14.277213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:14.277863 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:14.278096 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:14.278716 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 01:34:14.279113 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":416,"threads_started":1}
I20260925 01:34:14.283454 14129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175f12557bb423a804b9731582c10e4" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35907 } }
I20260925 01:34:14.284003 14129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:14.284130 14129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8175f12557bb423a804b9731582c10e4, State: Initialized, Role: FOLLOWER
I20260925 01:34:14.284827 14129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175f12557bb423a804b9731582c10e4" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35907 } }
I20260925 01:34:14.285110 14129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:14.285287 14129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:14.285487 14129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:14.288341 14129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175f12557bb423a804b9731582c10e4" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35907 } }
I20260925 01:34:14.288942 14129 leader_election.cc:302] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8175f12557bb423a804b9731582c10e4; no voters:
I20260925 01:34:14.289701 14129 leader_election.cc:288] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:14.289847 14134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:14.290688 14134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [term 1 LEADER]: Becoming Leader. State: Replica: 8175f12557bb423a804b9731582c10e4, State: Running, Role: LEADER
I20260925 01:34:14.291317 14134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175f12557bb423a804b9731582c10e4" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35907 } }
I20260925 01:34:14.292212 14129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:14.294363 14136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8175f12557bb423a804b9731582c10e4. Latest consensus state: current_term: 1 leader_uuid: "8175f12557bb423a804b9731582c10e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175f12557bb423a804b9731582c10e4" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35907 } } }
I20260925 01:34:14.294921 14136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:14.294929 14135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8175f12557bb423a804b9731582c10e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8175f12557bb423a804b9731582c10e4" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35907 } } }
I20260925 01:34:14.295547 14135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:14.297705 14140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:14.303294 14140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:14.315245 14140 catalog_manager.cc:1347] Generated new cluster ID: aa8785037eb14a2b840cf131d2573109
I20260925 01:34:14.315430 14140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:14.335470 14140 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:14.337265 14140 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:14.355280 14140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4: Generated new TSK 0
I20260925 01:34:14.356231 14140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:34:14.439092 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35907
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:14.795084 14155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:14.795380 14155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:14.795475 14155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:14.803021 14155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:14.803136 14155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:14.803249 14155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:34:14.812983 14155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:14.814288 14155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:14.815636 14155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:14.829526 14163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:14.829766 14160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:14.829744 14161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:14.831275 14155 server_base.cc:1034] running on GCE node
I20260925 01:34:14.832134 14155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:14.833437 14155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:14.834753 14155 hybrid_clock.cc:648] HybridClock initialized: now 1790300054834706 us; error 74 us; skew 500 ppm
I20260925 01:34:14.835260 14155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:14 dist-test-slave-h6hf krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300054, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:14.839282 14155 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:34:14.841192 14155 webserver.cc:460] Webserver started at http://127.27.160.1:42221/ using document root <none> and password file <none>
I20260925 01:34:14.842000 14155 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:14.842163 14155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:14.842519 14155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:14.845798 14155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "21aad6301edf4672bb677f25ab17826d"
format_stamp: "Formatted at 2026-09-25 01:34:14 on dist-test-slave-h6hf"
server_key: "ea6f4b098300d885852ad3c3f0adc723"
server_key_iv: "2f8b65e06e659b6afe1e026b24c0b482"
server_key_version: "encryptionkey@0"
I20260925 01:34:14.846634 14155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "21aad6301edf4672bb677f25ab17826d"
format_stamp: "Formatted at 2026-09-25 01:34:14 on dist-test-slave-h6hf"
server_key: "ea6f4b098300d885852ad3c3f0adc723"
server_key_iv: "2f8b65e06e659b6afe1e026b24c0b482"
server_key_version: "encryptionkey@0"
I20260925 01:34:14.852365 14155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:34:14.856783 14170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:14.858520 14155 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:34:14.858704 14155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "21aad6301edf4672bb677f25ab17826d"
format_stamp: "Formatted at 2026-09-25 01:34:14 on dist-test-slave-h6hf"
server_key: "ea6f4b098300d885852ad3c3f0adc723"
server_key_iv: "2f8b65e06e659b6afe1e026b24c0b482"
server_key_version: "encryptionkey@0"
I20260925 01:34:14.858914 14155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:14.875244 14155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:14.880016 14155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:14.880308 14155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:14.881533 14155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:14.883024 14155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:14.883106 14155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:14.883220 14155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:14.883282 14155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:14.921311 14155 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:37645
I20260925 01:34:14.921416 14282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:37645 every 8 connection(s)
I20260925 01:34:14.922760 14155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:34:14.931872 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 14155
I20260925 01:34:14.932090 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:34:14.932605 28288 external_mini_cluster.cc:1428] Setting key c0456123a92af2afaf00f9e9da87ed09
Sep 25 01:34:14 dist-test-slave-h6hf krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300054, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:14.948392 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:14.925541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:48703 (local address 127.27.160.62:35907)
0925 01:34:14.926068 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:14.926073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:14.927034 (+ 961us) server_negotiation.cc:407] Connection header received
0925 01:34:14.929112 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:14.929136 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:14.929236 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:14.929371 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:14.930989 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:14.932072 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:14.934253 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:14.934941 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:14.935120 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:14.939685 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:14.939696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:14.939704 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:14.939758 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:14.944033 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:14.945058 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:14.945068 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:14.945075 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:14.945316 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:14.945928 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:14.945936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:14.945942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:14.946417 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:14.946583 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:14.947525 (+ 942us) server_negotiation.cc:299] Negotiation successful
0925 01:34:14.947838 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":317,"threads_started":1}
I20260925 01:34:14.949317 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:14.926042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:35907 (local address 127.27.160.1:48703)
0925 01:34:14.926836 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:14.926886 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:14.928809 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:14.929623 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:14.929634 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:14.930176 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:14.930802 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:14.930821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:14.932288 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:14.932293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:14.934074 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:14.934092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:14.935266 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:14.935273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:14.935520 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:14.936365 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:14.936392 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:14.939353 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:14.944277 (+ 4924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:14.944287 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:14.944305 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:14.944885 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:14.945485 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:14.945490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:14.945493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:14.945772 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:14.946615 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:14.946623 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:14.947035 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 01:34:14.947364 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 01:34:14.947893 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":306,"threads_started":1}
I20260925 01:34:14.951437 14283 heartbeater.cc:347] Connected to a master server at 127.27.160.62:35907
I20260925 01:34:14.951931 14283 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:14.953073 14283 heartbeater.cc:511] Master 127.27.160.62:35907 requested a full tablet report, sending...
I20260925 01:34:14.955935 14094 ts_manager.cc:194] Registered new tserver with Master: 21aad6301edf4672bb677f25ab17826d (127.27.160.1:37645)
I20260925 01:34:14.957784 14094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:48703
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:34:15.001966 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35907
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:15.354828 14289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:15.355122 14289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:15.355216 14289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:15.363072 14289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:15.363209 14289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:15.363396 14289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:34:15.374193 14289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:15.375615 14289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:15.376991 14289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:15.391000 14294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:15.391088 14295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:15.391063 14297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:15.392886 14289 server_base.cc:1034] running on GCE node
I20260925 01:34:15.393795 14289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:15.394876 14289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:15.396138 14289 hybrid_clock.cc:648] HybridClock initialized: now 1790300055396094 us; error 37 us; skew 500 ppm
I20260925 01:34:15.396479 14289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:15 dist-test-slave-h6hf krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300055, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:15.400256 14289 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:34:15.401957 14289 webserver.cc:460] Webserver started at http://127.27.160.2:35987/ using document root <none> and password file <none>
I20260925 01:34:15.402678 14289 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:15.402788 14289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:15.403092 14289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:15.406288 14289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "f1688068ed0540049ff5ae82f81a0aa1"
format_stamp: "Formatted at 2026-09-25 01:34:15 on dist-test-slave-h6hf"
server_key: "86f6ad631b922a4e0ac506098648c88e"
server_key_iv: "3a7d168e722989534d33638071d37109"
server_key_version: "encryptionkey@0"
I20260925 01:34:15.407047 14289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "f1688068ed0540049ff5ae82f81a0aa1"
format_stamp: "Formatted at 2026-09-25 01:34:15 on dist-test-slave-h6hf"
server_key: "86f6ad631b922a4e0ac506098648c88e"
server_key_iv: "3a7d168e722989534d33638071d37109"
server_key_version: "encryptionkey@0"
I20260925 01:34:15.412760 14289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 01:34:15.416810 14304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:15.418604 14289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:15.418869 14289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "f1688068ed0540049ff5ae82f81a0aa1"
format_stamp: "Formatted at 2026-09-25 01:34:15 on dist-test-slave-h6hf"
server_key: "86f6ad631b922a4e0ac506098648c88e"
server_key_iv: "3a7d168e722989534d33638071d37109"
server_key_version: "encryptionkey@0"
I20260925 01:34:15.419080 14289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:15.437471 14289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:15.442793 14289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:15.443153 14289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:15.444260 14289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:15.445870 14289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:15.445948 14289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:15.446091 14289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:15.446146 14289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:15.483683 14289 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:45953
I20260925 01:34:15.483747 14416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:45953 every 8 connection(s)
I20260925 01:34:15.485323 14289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:34:15.491472 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 14289
I20260925 01:34:15.491683 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:34:15.492249 28288 external_mini_cluster.cc:1428] Setting key acdc874931b8006420ef2c23ac62e2a4
Sep 25 01:34:15 dist-test-slave-h6hf krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300055, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:15.509433 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:15.488419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:48537 (local address 127.27.160.62:35907)
0925 01:34:15.489014 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:15.489022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:15.490346 (+ 1324us) server_negotiation.cc:407] Connection header received
0925 01:34:15.492640 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:15.492648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:15.492766 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:15.492911 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:15.495029 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:15.495765 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:15.497611 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:15.498271 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:15.498433 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:15.502655 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:15.502661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:15.502664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:15.502701 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:15.505555 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:15.506605 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:15.506609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:15.506611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:15.506822 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:15.507389 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:15.507393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:15.507395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:15.507738 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:15.507864 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:15.508894 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0925 01:34:15.509083 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":312,"threads_started":1}
I20260925 01:34:15.511228 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:15.489046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:35907 (local address 127.27.160.2:48537)
0925 01:34:15.490150 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:15.490194 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:15.492213 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:15.493252 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:15.493266 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:15.494036 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:15.494833 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:15.494855 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:15.495960 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:15.495967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:15.497418 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:15.497434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:15.498558 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:15.498567 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:15.498861 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:15.499755 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:15.499781 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:15.502324 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:15.505719 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:15.505732 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:15.505755 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:15.506465 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:15.506952 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:15.506961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:15.506968 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:15.507268 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:15.507886 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:15.507898 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:15.508444 (+ 546us) client_negotiation.cc:769] Sending connection context
0925 01:34:15.508796 (+ 352us) client_negotiation.cc:240] Negotiation successful
0925 01:34:15.509499 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":423,"threads_started":1}
I20260925 01:34:15.512861 14417 heartbeater.cc:347] Connected to a master server at 127.27.160.62:35907
I20260925 01:34:15.513264 14417 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:15.514057 14417 heartbeater.cc:511] Master 127.27.160.62:35907 requested a full tablet report, sending...
I20260925 01:34:15.515805 14094 ts_manager.cc:194] Registered new tserver with Master: f1688068ed0540049ff5ae82f81a0aa1 (127.27.160.2:45953)
I20260925 01:34:15.516682 14094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:48537
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:34:15.559154 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35907
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:15.919059 14423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:15.919394 14423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:15.919574 14423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:15.931100 14423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:15.931234 14423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:15.931419 14423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:34:15.942513 14423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:35907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:15.943944 14423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:15.945459 14423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:15.960462 14431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:15.960496 14429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:15.960781 14428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:15.962175 14423 server_base.cc:1034] running on GCE node
I20260925 01:34:15.962711 14283 heartbeater.cc:503] Master 127.27.160.62:35907 was elected leader, sending a full tablet report...
I20260925 01:34:15.963189 14423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:15.964600 14423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:15.965884 14423 hybrid_clock.cc:648] HybridClock initialized: now 1790300055965808 us; error 62 us; skew 500 ppm
I20260925 01:34:15.966323 14423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:15 dist-test-slave-h6hf krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300055, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:15.970396 14423 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:34:15.972316 14423 webserver.cc:460] Webserver started at http://127.27.160.3:39651/ using document root <none> and password file <none>
I20260925 01:34:15.973112 14423 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:15.973347 14423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:15.973737 14423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:15.977224 14423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "44604509442546f19ad6f641c1248b9e"
format_stamp: "Formatted at 2026-09-25 01:34:15 on dist-test-slave-h6hf"
server_key: "2f437ebb3438219857232915e5f939ac"
server_key_iv: "6eaafc9c29f9cc0fc20dec17100e72aa"
server_key_version: "encryptionkey@0"
I20260925 01:34:15.978139 14423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "44604509442546f19ad6f641c1248b9e"
format_stamp: "Formatted at 2026-09-25 01:34:15 on dist-test-slave-h6hf"
server_key: "2f437ebb3438219857232915e5f939ac"
server_key_iv: "6eaafc9c29f9cc0fc20dec17100e72aa"
server_key_version: "encryptionkey@0"
I20260925 01:34:15.984164 14423 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 01:34:15.988468 14438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:15.990126 14423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:15.990335 14423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "44604509442546f19ad6f641c1248b9e"
format_stamp: "Formatted at 2026-09-25 01:34:15 on dist-test-slave-h6hf"
server_key: "2f437ebb3438219857232915e5f939ac"
server_key_iv: "6eaafc9c29f9cc0fc20dec17100e72aa"
server_key_version: "encryptionkey@0"
I20260925 01:34:15.990563 14423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:16.011456 14423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:16.016458 14423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:16.016876 14423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:16.018003 14423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:16.019526 14423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:16.019614 14423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:16.019797 14423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:16.019855 14423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:16.056596 14423 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:44101
I20260925 01:34:16.056636 14550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:44101 every 8 connection(s)
I20260925 01:34:16.057991 14423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:34:16.067843 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 14423
I20260925 01:34:16.068051 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:34:16.068542 28288 external_mini_cluster.cc:1428] Setting key 056954911e120bb27d09033fcfd31386
Sep 25 01:34:16 dist-test-slave-h6hf krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300055, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:16.080760 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.061019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:53813 (local address 127.27.160.62:35907)
0925 01:34:16.061533 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.061538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.062543 (+ 1005us) server_negotiation.cc:407] Connection header received
0925 01:34:16.064623 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.064629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.064743 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.064881 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:16.066588 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.067406 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.069394 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.069975 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.070129 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.074481 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:16.074487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:16.074490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:16.074526 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:16.077446 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.078306 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.078310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.078312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.078442 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.078985 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.078990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.078992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.079310 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:16.079444 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.080245 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.080440 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
I20260925 01:34:16.081845 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.061622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:35907 (local address 127.27.160.3:53813)
0925 01:34:16.062365 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:16.062412 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:16.064340 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:16.065211 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:16.065220 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:16.065757 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:16.066399 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.066420 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.067593 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.067597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.069203 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.069221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.070312 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.070321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.070706 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.071581 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:16.071617 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:16.074247 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:16.077621 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:16.077634 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:16.077660 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:16.078182 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:16.078582 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:16.078586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:16.078588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:16.078815 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:16.079451 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:16.079457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:16.079892 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 01:34:16.080145 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 01:34:16.080586 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":297,"threads_started":1}
I20260925 01:34:16.083565 14551 heartbeater.cc:347] Connected to a master server at 127.27.160.62:35907
I20260925 01:34:16.083984 14551 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:16.084887 14551 heartbeater.cc:511] Master 127.27.160.62:35907 requested a full tablet report, sending...
I20260925 01:34:16.086447 14094 ts_manager.cc:194] Registered new tserver with Master: 44604509442546f19ad6f641c1248b9e (127.27.160.3:44101)
I20260925 01:34:16.087239 14094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:53813
I20260925 01:34:16.097573 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:34:16.115808 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.102301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.27.160.62:35907)
0925 01:34:16.102649 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.102656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.102905 (+ 249us) server_negotiation.cc:407] Connection header received
0925 01:34:16.103036 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.103040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.103110 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.103209 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:16.104649 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.105528 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.106885 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.107317 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.107420 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.110084 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:16.110095 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:16.110098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:16.110136 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:16.112878 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.113736 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.113741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.113743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.113910 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.114311 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.114314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.114316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.114691 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:16.114920 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.115229 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.115372 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:34:16.120476 14094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59122:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:34:16.127645 14094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:34:16.156556 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.141625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.27.160.1:37645)
0925 01:34:16.142759 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.142772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.143477 (+ 705us) server_negotiation.cc:407] Connection header received
0925 01:34:16.143855 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.143868 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.144219 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.144421 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:16.145796 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.148586 (+ 2790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.153318 (+ 4732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.154881 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.155001 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.155275 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.155978 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.156176 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":746,"threads_started":1}
I20260925 01:34:16.156612 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.142020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:37645 (local address 127.0.0.1:36734)
0925 01:34:16.143340 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:16.143364 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:16.143609 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:16.144892 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:16.144904 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:16.144946 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:16.145526 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.145544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.148861 (+ 3317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.148870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.152276 (+ 3406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.152302 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.155478 (+ 3176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.155483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.155706 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.155764 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 01:34:16.155902 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 01:34:16.156138 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"thread_start_us":10626,"threads_started":1}
I20260925 01:34:16.160710 14218 tablet_service.cc:1467] Processing CreateTablet for tablet f17bb62f06064879af786401d91b6970 (DEFAULT_TABLE table=test-table [id=333536b9d85c40478733bc5f9feed91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:16.162616 14218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17bb62f06064879af786401d91b6970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:16.164556 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.141065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.27.160.2:45953)
0925 01:34:16.155531 (+ 14466us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.155539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.155566 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:34:16.155669 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.155676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.155925 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.156843 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:16.157253 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.158761 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.161822 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.163369 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.163502 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.163697 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.164015 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.164198 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14296,"thread_start_us":14164,"threads_started":1}
I20260925 01:34:16.164595 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.140872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:45953 (local address 127.0.0.1:34426)
0925 01:34:16.141708 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:16.141754 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:16.141965 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:16.156141 (+ 14176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:16.156145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:16.156183 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:16.156576 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.156597 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.158900 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.158909 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.161621 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.161639 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.163598 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.163605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.163792 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.163832 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 01:34:16.163943 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:34:16.164100 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":101,"thread_start_us":366,"threads_started":1}
I20260925 01:34:16.165292 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.143037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.27.160.3:44101)
0925 01:34:16.155454 (+ 12417us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.155464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.155504 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:34:16.155620 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.155629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.155904 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.156137 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:16.157244 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.158792 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.161822 (+ 3030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.163369 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.163502 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.163741 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.164722 (+ 981us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.164887 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12206,"thread_start_us":12429,"threads_started":1}
I20260925 01:34:16.165410 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.142334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:44101 (local address 127.0.0.1:39074)
0925 01:34:16.152952 (+ 10618us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:16.152974 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:16.153103 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:16.156425 (+ 3322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:16.156430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:16.156462 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:16.156973 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.156986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.159004 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.159014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.161621 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.161639 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.163873 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.163877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.164125 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.164160 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:34:16.164253 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 01:34:16.164396 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10462,"mutex_wait_us":193,"thread_start_us":1029,"threads_started":1}
I20260925 01:34:16.168869 14486 tablet_service.cc:1467] Processing CreateTablet for tablet f17bb62f06064879af786401d91b6970 (DEFAULT_TABLE table=test-table [id=333536b9d85c40478733bc5f9feed91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:16.169051 14352 tablet_service.cc:1467] Processing CreateTablet for tablet f17bb62f06064879af786401d91b6970 (DEFAULT_TABLE table=test-table [id=333536b9d85c40478733bc5f9feed91e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:16.170634 14352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17bb62f06064879af786401d91b6970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:16.171908 14486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f17bb62f06064879af786401d91b6970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:16.183080 14570 tablet_bootstrap.cc:492] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Bootstrap starting.
I20260925 01:34:16.185261 14571 tablet_bootstrap.cc:492] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Bootstrap starting.
I20260925 01:34:16.187184 14572 tablet_bootstrap.cc:492] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Bootstrap starting.
I20260925 01:34:16.187222 14570 tablet_bootstrap.cc:654] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:16.188628 14570 log.cc:824] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:16.189479 14571 tablet_bootstrap.cc:654] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:16.190914 14572 tablet_bootstrap.cc:654] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:16.190959 14571 log.cc:824] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:16.192188 14572 log.cc:824] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:16.192965 14570 tablet_bootstrap.cc:492] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: No bootstrap required, opened a new log
I20260925 01:34:16.193363 14570 ts_tablet_manager.cc:1397] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260925 01:34:16.194135 14571 tablet_bootstrap.cc:492] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: No bootstrap required, opened a new log
I20260925 01:34:16.194453 14571 ts_tablet_manager.cc:1397] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260925 01:34:16.195415 14572 tablet_bootstrap.cc:492] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: No bootstrap required, opened a new log
I20260925 01:34:16.195683 14572 ts_tablet_manager.cc:1397] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:34:16.203025 14571 raft_consensus.cc:348] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.203465 14571 raft_consensus.cc:374] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:16.203614 14571 raft_consensus.cc:729] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44604509442546f19ad6f641c1248b9e, State: Initialized, Role: FOLLOWER
I20260925 01:34:16.204293 14571 consensus_queue.cc:260] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.204733 14570 raft_consensus.cc:348] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.205304 14570 raft_consensus.cc:374] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:16.205418 14570 raft_consensus.cc:729] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21aad6301edf4672bb677f25ab17826d, State: Initialized, Role: FOLLOWER
I20260925 01:34:16.205570 14551 heartbeater.cc:503] Master 127.27.160.62:35907 was elected leader, sending a full tablet report...
I20260925 01:34:16.206171 14571 ts_tablet_manager.cc:1428] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 01:34:16.206175 14570 consensus_queue.cc:260] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.207902 14570 ts_tablet_manager.cc:1428] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 01:34:16.208448 14572 raft_consensus.cc:348] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.208997 14572 raft_consensus.cc:374] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:16.209189 14572 raft_consensus.cc:729] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1688068ed0540049ff5ae82f81a0aa1, State: Initialized, Role: FOLLOWER
I20260925 01:34:16.209883 14572 consensus_queue.cc:260] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.212042 14417 heartbeater.cc:503] Master 127.27.160.62:35907 was elected leader, sending a full tablet report...
I20260925 01:34:16.212819 14572 ts_tablet_manager.cc:1428] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260925 01:34:16.237776 14418 tablet.cc:2369] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:34:16.309953 14552 tablet.cc:2369] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:34:16.425657 14284 tablet.cc:2369] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:34:16.497298 14576 raft_consensus.cc:482] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:34:16.497690 14576 raft_consensus.cc:504] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.499454 14576 leader_election.cc:288] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1688068ed0540049ff5ae82f81a0aa1 (127.27.160.2:45953), 21aad6301edf4672bb677f25ab17826d (127.27.160.1:37645)
I20260925 01:34:16.502086 14577 raft_consensus.cc:482] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:34:16.502406 14577 raft_consensus.cc:504] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.504101 14577 leader_election.cc:288] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1688068ed0540049ff5ae82f81a0aa1 (127.27.160.2:45953), 44604509442546f19ad6f641c1248b9e (127.27.160.3:44101)
I20260925 01:34:16.507491 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.499828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:37645 (local address 127.27.160.3:51381)
0925 01:34:16.500180 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:16.500203 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:16.500369 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:16.500795 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:16.500802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:16.500834 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:16.501053 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.501068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.502538 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.502544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.504753 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.504769 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.506692 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.506701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.506929 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.506964 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:34:16.507056 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 01:34:16.507214 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260925 01:34:16.507588 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.499942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:53293 (local address 127.27.160.2:45953)
0925 01:34:16.500154 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.500161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.500760 (+ 599us) server_negotiation.cc:407] Connection header received
0925 01:34:16.500910 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.500915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.500996 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.501156 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:16.501779 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.502716 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.505139 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.506468 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.506589 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.506664 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.507200 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.507335 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:34:16.508301 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.500074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:51381 (local address 127.27.160.1:37645)
0925 01:34:16.500279 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.500286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.500309 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:34:16.500549 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.500555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.500654 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.500748 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:16.501224 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.502399 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.504969 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.506313 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.506482 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.506583 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.507904 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.508049 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:34:16.508441 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.499933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:45953 (local address 127.27.160.3:53293)
0925 01:34:16.500601 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:16.500618 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:16.500775 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:16.501188 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:16.501193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:16.501218 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:16.501569 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.501581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.502892 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.502898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.504934 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.504947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.506838 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.506844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.507008 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.507050 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 01:34:16.507928 (+ 878us) client_negotiation.cc:240] Negotiation successful
0925 01:34:16.508067 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":1,"thread_start_us":370,"threads_started":1}
I20260925 01:34:16.509739 14238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17bb62f06064879af786401d91b6970" candidate_uuid: "44604509442546f19ad6f641c1248b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21aad6301edf4672bb677f25ab17826d" is_pre_election: true
I20260925 01:34:16.510288 14238 raft_consensus.cc:2454] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44604509442546f19ad6f641c1248b9e in term 0.
I20260925 01:34:16.510279 14372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17bb62f06064879af786401d91b6970" candidate_uuid: "44604509442546f19ad6f641c1248b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1688068ed0540049ff5ae82f81a0aa1" is_pre_election: true
I20260925 01:34:16.510882 14372 raft_consensus.cc:2454] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44604509442546f19ad6f641c1248b9e in term 0.
I20260925 01:34:16.511682 14439 leader_election.cc:302] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21aad6301edf4672bb677f25ab17826d, 44604509442546f19ad6f641c1248b9e; no voters:
I20260925 01:34:16.512131 14576 raft_consensus.cc:2781] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:34:16.512270 14576 raft_consensus.cc:482] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:34:16.512369 14576 raft_consensus.cc:3037] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:16.515765 14576 raft_consensus.cc:504] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.517014 14372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17bb62f06064879af786401d91b6970" candidate_uuid: "44604509442546f19ad6f641c1248b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1688068ed0540049ff5ae82f81a0aa1"
I20260925 01:34:16.517045 14238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17bb62f06064879af786401d91b6970" candidate_uuid: "44604509442546f19ad6f641c1248b9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21aad6301edf4672bb677f25ab17826d"
I20260925 01:34:16.517295 14372 raft_consensus.cc:3037] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:16.517305 14238 raft_consensus.cc:3037] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:16.520910 14238 raft_consensus.cc:2454] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44604509442546f19ad6f641c1248b9e in term 1.
I20260925 01:34:16.520915 14372 raft_consensus.cc:2454] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44604509442546f19ad6f641c1248b9e in term 1.
I20260925 01:34:16.521165 14576 leader_election.cc:288] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [CANDIDATE]: Term 1 election: Requested vote from peers f1688068ed0540049ff5ae82f81a0aa1 (127.27.160.2:45953), 21aad6301edf4672bb677f25ab17826d (127.27.160.1:37645)
I20260925 01:34:16.521471 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.504674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:42453 (local address 127.27.160.2:45953)
0925 01:34:16.507819 (+ 3145us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.507824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.513523 (+ 5699us) server_negotiation.cc:407] Connection header received
0925 01:34:16.513710 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.513715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.513825 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.513919 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:16.514764 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.515720 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.518666 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.520043 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.520196 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.520283 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.520929 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.521092 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3046,"thread_start_us":8049,"threads_started":1}
I20260925 01:34:16.521615 14441 leader_election.cc:302] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44604509442546f19ad6f641c1248b9e, f1688068ed0540049ff5ae82f81a0aa1; no voters:
I20260925 01:34:16.522128 14576 raft_consensus.cc:2781] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:16.522325 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.513796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:48271 (local address 127.27.160.3:44101)
0925 01:34:16.513988 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.513997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.514873 (+ 876us) server_negotiation.cc:407] Connection header received
0925 01:34:16.515005 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.515009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.515093 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.515198 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:16.515863 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.517021 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.519674 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.520959 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.521101 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.521198 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.521843 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.521986 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:34:16.522413 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.504307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:45953 (local address 127.27.160.1:42453)
0925 01:34:16.513409 (+ 9102us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:16.513430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:16.513569 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:16.514095 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:16.514103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:16.514137 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:16.514619 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.514629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.515928 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.515935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.518450 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.518467 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.520492 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.520513 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.520709 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.520752 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 01:34:16.521831 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0925 01:34:16.521986 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8948,"mutex_wait_us":71,"thread_start_us":8726,"threads_started":1}
I20260925 01:34:16.523130 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.514012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:44101 (local address 127.27.160.1:48271)
0925 01:34:16.514751 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:16.514769 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:16.514904 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:16.515225 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:16.515230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:16.515254 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:16.515674 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.515688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.517230 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.517236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.519477 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:16.519492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.521408 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.521415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:16.521626 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.521666 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 01:34:16.522166 (+ 500us) client_negotiation.cc:240] Negotiation successful
0925 01:34:16.522715 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":1,"thread_start_us":346,"threads_started":1}
I20260925 01:34:16.523095 14372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17bb62f06064879af786401d91b6970" candidate_uuid: "21aad6301edf4672bb677f25ab17826d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1688068ed0540049ff5ae82f81a0aa1" is_pre_election: true
I20260925 01:34:16.523458 14372 raft_consensus.cc:2379] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 21aad6301edf4672bb677f25ab17826d in current term 1: Already voted for candidate 44604509442546f19ad6f641c1248b9e in this term.
I20260925 01:34:16.524086 14576 raft_consensus.cc:686] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 1 LEADER]: Becoming Leader. State: Replica: 44604509442546f19ad6f641c1248b9e, State: Running, Role: LEADER
I20260925 01:34:16.524013 14506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f17bb62f06064879af786401d91b6970" candidate_uuid: "21aad6301edf4672bb677f25ab17826d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44604509442546f19ad6f641c1248b9e" is_pre_election: true
I20260925 01:34:16.524788 14576 consensus_queue.cc:237] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } }
I20260925 01:34:16.525147 14173 leader_election.cc:302] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21aad6301edf4672bb677f25ab17826d; no voters: 44604509442546f19ad6f641c1248b9e, f1688068ed0540049ff5ae82f81a0aa1
I20260925 01:34:16.525586 14577 raft_consensus.cc:2726] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 01:34:16.530010 14092 catalog_manager.cc:5495] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 44604509442546f19ad6f641c1248b9e (127.27.160.3). New cstate: current_term: 1 leader_uuid: "44604509442546f19ad6f641c1248b9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44604509442546f19ad6f641c1248b9e" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 44101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 } health_report { overall_health: UNKNOWN } } }
I20260925 01:34:16.631609 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.625086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.27.160.3:44101)
0925 01:34:16.625353 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.625359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.625816 (+ 457us) server_negotiation.cc:407] Connection header received
0925 01:34:16.625954 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.625958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.626055 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.626205 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:34:16.626806 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.627570 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.629270 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.629760 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.629873 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.630437 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:34:16.631010 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:34:16.631145 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.631327 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.631369 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 01:34:16.641157 14372 raft_consensus.cc:1261] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 1 FOLLOWER]: Refusing update from remote peer 44604509442546f19ad6f641c1248b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:16.641206 14238 raft_consensus.cc:1261] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 1 FOLLOWER]: Refusing update from remote peer 44604509442546f19ad6f641c1248b9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:16.642328 14583 consensus_queue.cc:1035] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1688068ed0540049ff5ae82f81a0aa1" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 45953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:16.642741 14576 consensus_queue.cc:1035] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "21aad6301edf4672bb677f25ab17826d" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:16.672664 14093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:16.673017 14093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59122:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:16.677629 14093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8175f12557bb423a804b9731582c10e4: Sending DeleteTablet for 3 replicas of tablet f17bb62f06064879af786401d91b6970
I20260925 01:34:16.679076 14218 tablet_service.cc:1514] Processing DeleteTablet for tablet f17bb62f06064879af786401d91b6970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:16 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:36734
I20260925 01:34:16.679488 14486 tablet_service.cc:1514] Processing DeleteTablet for tablet f17bb62f06064879af786401d91b6970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:16 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:39074
I20260925 01:34:16.679533 14352 tablet_service.cc:1514] Processing DeleteTablet for tablet f17bb62f06064879af786401d91b6970 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:16 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:34426
I20260925 01:34:16.681466 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 14155
I20260925 01:34:16.691706 14597 tablet_replica.cc:330] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: stopping tablet replica
I20260925 01:34:16.692838 14597 raft_consensus.cc:2229] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:16.694003 14597 raft_consensus.cc:2258] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:16.694911 14599 tablet_replica.cc:330] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: stopping tablet replica
I20260925 01:34:16.695389 14599 raft_consensus.cc:2229] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:16.696043 14599 raft_consensus.cc:2258] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:16.696524 14597 ts_tablet_manager.cc:1905] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:16.696918 14598 tablet_replica.cc:330] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: stopping tablet replica
I20260925 01:34:16.698542 14599 ts_tablet_manager.cc:1905] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:16.699529 14598 raft_consensus.cc:2229] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 1 LEADER]: Raft consensus shutting down.
Sep 25 01:34:16 dist-test-slave-h6hf krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:34:16.705505 14597 ts_tablet_manager.cc:1918] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:16.705763 14597 log.cc:1196] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/f17bb62f06064879af786401d91b6970
I20260925 01:34:16.705858 14598 raft_consensus.cc:2258] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:16.706696 14597 ts_tablet_manager.cc:1939] T f17bb62f06064879af786401d91b6970 P 21aad6301edf4672bb677f25ab17826d: Deleting consensus metadata
I20260925 01:34:16.708325 14598 ts_tablet_manager.cc:1905] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:16.709111 14078 catalog_manager.cc:4841] TS 21aad6301edf4672bb677f25ab17826d (127.27.160.1:37645): tablet f17bb62f06064879af786401d91b6970 (table test-table [id=333536b9d85c40478733bc5f9feed91e]) successfully deleted
I20260925 01:34:16.711277 14599 ts_tablet_manager.cc:1918] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:16.711520 14599 log.cc:1196] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/f17bb62f06064879af786401d91b6970
I20260925 01:34:16.712122 14599 ts_tablet_manager.cc:1939] T f17bb62f06064879af786401d91b6970 P f1688068ed0540049ff5ae82f81a0aa1: Deleting consensus metadata
I20260925 01:34:16.712965 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.683220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.27.160.1:37645)
0925 01:34:16.683463 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.683471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.688065 (+ 4594us) server_negotiation.cc:407] Connection header received
0925 01:34:16.689295 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.689305 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.689469 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.689564 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:16.694134 (+ 4570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.695424 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.697545 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.698261 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.698365 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.704239 (+ 5874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:16.704258 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:16.704278 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:16.704326 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:16.708755 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.710054 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.710064 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.710068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.710237 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.710731 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.710738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.710743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.711232 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:16.711577 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.712306 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.712535 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:34:16.714272 14080 catalog_manager.cc:4841] TS f1688068ed0540049ff5ae82f81a0aa1 (127.27.160.2:45953): tablet f17bb62f06064879af786401d91b6970 (table test-table [id=333536b9d85c40478733bc5f9feed91e]) successfully deleted
I20260925 01:34:16.714789 14278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:16.716300 14598 ts_tablet_manager.cc:1918] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:16.716621 14598 log.cc:1196] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/f17bb62f06064879af786401d91b6970
I20260925 01:34:16.717425 14598 ts_tablet_manager.cc:1939] T f17bb62f06064879af786401d91b6970 P 44604509442546f19ad6f641c1248b9e: Deleting consensus metadata
I20260925 01:34:16.719691 14080 catalog_manager.cc:4841] TS 44604509442546f19ad6f641c1248b9e (127.27.160.3:44101): tablet f17bb62f06064879af786401d91b6970 (table test-table [id=333536b9d85c40478733bc5f9feed91e]) successfully deleted
I20260925 01:34:16.808653 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 14155
I20260925 01:34:16.828933 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 14289
Sep 25 01:34:16 dist-test-slave-h6hf krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:34:16.847064 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.829877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.27.160.2:45953)
0925 01:34:16.830065 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.830070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.830088 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:34:16.830359 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.830364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.830436 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.830537 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:16.832271 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.833057 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.834643 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.835094 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.835255 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.840021 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:16.840040 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:16.840057 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:16.840110 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:16.843478 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.844388 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.844399 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.844401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.844556 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.845026 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.845033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.845037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.845647 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:16.846025 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.846390 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.846658 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:34:16.847774 14412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:16.943230 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 14289
I20260925 01:34:16.961045 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 14423
Sep 25 01:34:16 dist-test-slave-h6hf krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:34:16.981540 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:16.962192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.27.160.3:44101)
0925 01:34:16.962393 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:16.962399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:16.962444 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:34:16.962515 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:16.962520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:16.962636 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:16.962733 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:16.964691 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.965835 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.967822 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:16.968348 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:16.968470 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:16.974644 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:16.974664 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:16.974683 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:16.974734 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:16.978110 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.979039 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.979047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.979050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.979217 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:16.979648 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:16.979658 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:16.979660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:16.980163 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:16.980544 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:16.980860 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 01:34:16.981183 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:34:16.982389 14546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:17.114022 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 14423
I20260925 01:34:17.130949 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 14062
I20260925 01:34:17.132194 14124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:17.249331 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 14062
2026-09-25T01:34:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4921 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T01:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:17Z Disabled control of system clock
I20260925 01:34:17.330265 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:41449
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:41449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:34:17.679301 14608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:17.679642 14608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:17.679736 14608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:17.687559 14608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:17.687705 14608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:17.687744 14608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:17.687795 14608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:17.687839 14608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:17.697803 14608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:41449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:17.699339 14608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:17.701491 14608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:17.714686 14613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:17.714596 14616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:17.714615 14614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:17.715888 14608 server_base.cc:1034] running on GCE node
I20260925 01:34:17.716732 14608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:17.718398 14608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:17.719723 14608 hybrid_clock.cc:648] HybridClock initialized: now 1790300057719645 us; error 77 us; skew 500 ppm
I20260925 01:34:17.720132 14608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:17.722888 14608 webserver.cc:460] Webserver started at http://127.27.160.62:46425/ using document root <none> and password file <none>
I20260925 01:34:17.723671 14608 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:17.723816 14608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:17.724164 14608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:17.727459 14608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "045600befc7443b0a7b73f7a197a1f03"
format_stamp: "Formatted at 2026-09-25 01:34:17 on dist-test-slave-h6hf"
server_key: "d31391b17384e6daee303b5e3931c33c"
server_key_iv: "51181dab0ad0ac226dd85608b4ba1758"
server_key_version: "encryptionkey@0"
I20260925 01:34:17.728343 14608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "045600befc7443b0a7b73f7a197a1f03"
format_stamp: "Formatted at 2026-09-25 01:34:17 on dist-test-slave-h6hf"
server_key: "d31391b17384e6daee303b5e3931c33c"
server_key_iv: "51181dab0ad0ac226dd85608b4ba1758"
server_key_version: "encryptionkey@0"
I20260925 01:34:17.734140 14608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:34:17.737985 14622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:17.739801 14608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:17.740021 14608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "045600befc7443b0a7b73f7a197a1f03"
format_stamp: "Formatted at 2026-09-25 01:34:17 on dist-test-slave-h6hf"
server_key: "d31391b17384e6daee303b5e3931c33c"
server_key_iv: "51181dab0ad0ac226dd85608b4ba1758"
server_key_version: "encryptionkey@0"
I20260925 01:34:17.740300 14608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:17.756299 14608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:17.757409 14608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:17.757755 14608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:17.778501 14608 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:41449
I20260925 01:34:17.778514 14673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:41449 every 8 connection(s)
I20260925 01:34:17.780134 14608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:17.784147 14674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:17.787499 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 14608
I20260925 01:34:17.787724 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:17.788204 28288 external_mini_cluster.cc:1428] Setting key f939bb9b59aeccf0c41a1174131be916
I20260925 01:34:17.797343 14674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03: Bootstrap starting.
I20260925 01:34:17.800809 14674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:17.802368 14674 log.cc:824] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:17.803151 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:17.790279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.27.160.62:41449)
0925 01:34:17.791289 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:17.791301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:17.791343 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:34:17.793195 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:17.793228 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:17.793765 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:17.794182 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:17.794784 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:17.796342 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:17.797903 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:17.798611 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:17.798838 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:17.800055 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:17.800064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:17.800081 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:17.800629 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:17.800736 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:17.800758 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:17.800893 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:17.801229 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 01:34:17.801641 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":342,"threads_started":1}
I20260925 01:34:17.805969 14674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03: No bootstrap required, opened a new log
I20260925 01:34:17.813482 14674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045600befc7443b0a7b73f7a197a1f03" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41449 } }
I20260925 01:34:17.813868 14674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:17.813997 14674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045600befc7443b0a7b73f7a197a1f03, State: Initialized, Role: FOLLOWER
I20260925 01:34:17.814612 14674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045600befc7443b0a7b73f7a197a1f03" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41449 } }
I20260925 01:34:17.814834 14674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:17.814932 14674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:17.815063 14674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:17.817690 14674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045600befc7443b0a7b73f7a197a1f03" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41449 } }
I20260925 01:34:17.818163 14674 leader_election.cc:302] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 045600befc7443b0a7b73f7a197a1f03; no voters:
I20260925 01:34:17.818792 14674 leader_election.cc:288] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:17.818957 14679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:17.819732 14679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [term 1 LEADER]: Becoming Leader. State: Replica: 045600befc7443b0a7b73f7a197a1f03, State: Running, Role: LEADER
I20260925 01:34:17.820622 14679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045600befc7443b0a7b73f7a197a1f03" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41449 } }
I20260925 01:34:17.821095 14674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:17.823630 14680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "045600befc7443b0a7b73f7a197a1f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045600befc7443b0a7b73f7a197a1f03" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41449 } } }
I20260925 01:34:17.824144 14680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:17.824093 14681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 045600befc7443b0a7b73f7a197a1f03. Latest consensus state: current_term: 1 leader_uuid: "045600befc7443b0a7b73f7a197a1f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "045600befc7443b0a7b73f7a197a1f03" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 41449 } } }
I20260925 01:34:17.824368 14681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:17.826229 14686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:17.831415 14686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:17.840956 14686 catalog_manager.cc:1347] Generated new cluster ID: c55a00fa4a1442129b2431525b8178ea
I20260925 01:34:17.841085 14686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:17.870370 14686 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:17.871958 14686 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:17.881774 14686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03: Generated new TSK 0
I20260925 01:34:17.882965 14686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:17.898423 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41449
--builtin_ntp_servers=127.27.160.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:34:18.256326 14698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:18.256632 14698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:18.256732 14698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:18.264218 14698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:18.264341 14698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:18.264451 14698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:34:18.274012 14698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:18.275315 14698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:18.277045 14698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:18.290977 14703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:18.291076 14704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:18.291028 14706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:18.294694 14698 server_base.cc:1034] running on GCE node
I20260925 01:34:18.295410 14698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:18.296473 14698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:18.297725 14698 hybrid_clock.cc:648] HybridClock initialized: now 1790300058297679 us; error 63 us; skew 500 ppm
I20260925 01:34:18.298138 14698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:18.300765 14698 webserver.cc:460] Webserver started at http://127.27.160.1:46847/ using document root <none> and password file <none>
I20260925 01:34:18.301633 14698 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:18.301781 14698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:18.302114 14698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:18.305320 14698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "0464c45b6b314f429816914ab2967170"
format_stamp: "Formatted at 2026-09-25 01:34:18 on dist-test-slave-h6hf"
server_key: "7c15a3b7645319b72b3b59dbf1cf70f0"
server_key_iv: "0eed5735067a3bb26063cb1329c95d30"
server_key_version: "encryptionkey@0"
I20260925 01:34:18.306157 14698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "0464c45b6b314f429816914ab2967170"
format_stamp: "Formatted at 2026-09-25 01:34:18 on dist-test-slave-h6hf"
server_key: "7c15a3b7645319b72b3b59dbf1cf70f0"
server_key_iv: "0eed5735067a3bb26063cb1329c95d30"
server_key_version: "encryptionkey@0"
I20260925 01:34:18.311951 14698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:34:18.315749 14712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:18.317282 14698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:18.317472 14698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "0464c45b6b314f429816914ab2967170"
format_stamp: "Formatted at 2026-09-25 01:34:18 on dist-test-slave-h6hf"
server_key: "7c15a3b7645319b72b3b59dbf1cf70f0"
server_key_iv: "0eed5735067a3bb26063cb1329c95d30"
server_key_version: "encryptionkey@0"
I20260925 01:34:18.317708 14698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:18.330794 14698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:18.331691 14698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:18.332020 14698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:18.333108 14698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:18.334681 14698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:18.334816 14698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:18.334945 14698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:18.335039 14698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:18.370105 14698 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:40989
I20260925 01:34:18.370198 14824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:40989 every 8 connection(s)
I20260925 01:34:18.371563 14698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:34:18.380023 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 14698
I20260925 01:34:18.380247 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:34:18.380787 28288 external_mini_cluster.cc:1428] Setting key 563f899d4e79339d011173f1dbe55ada
I20260925 01:34:18.383929 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41449
--builtin_ntp_servers=127.27.160.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:34:18.386157 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:18.374416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:58943 (local address 127.27.160.62:41449)
0925 01:34:18.374887 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:18.374894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:18.376300 (+ 1406us) server_negotiation.cc:407] Connection header received
0925 01:34:18.378391 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:18.378396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:18.378482 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:18.378586 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:18.380001 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:18.380988 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:18.383099 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:18.383519 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:18.383620 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:18.385371 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:18.385376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:18.385379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:18.385398 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:18.385439 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:18.385451 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:18.385524 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:18.385816 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 01:34:18.385898 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260925 01:34:18.387702 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:18.375072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41449 (local address 127.27.160.1:58943)
0925 01:34:18.376089 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:18.376135 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:18.378089 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:18.378869 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:18.378879 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:18.378981 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:18.379785 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:18.379815 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:18.381197 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:18.381204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:18.382957 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:18.382972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:18.383776 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:18.383784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:18.384137 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:18.384944 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:18.384950 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:18.385025 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:18.385172 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:18.385552 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:18.385565 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:18.385576 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:34:18.385782 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 01:34:18.386230 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":318,"threads_started":1}
I20260925 01:34:18.389421 14825 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41449
I20260925 01:34:18.389765 14825 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:18.390755 14825 heartbeater.cc:511] Master 127.27.160.62:41449 requested a full tablet report, sending...
I20260925 01:34:18.393177 14639 ts_manager.cc:194] Registered new tserver with Master: 0464c45b6b314f429816914ab2967170 (127.27.160.1:40989)
I20260925 01:34:18.395252 14639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.1:58943
W20260925 01:34:18.730643 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:18.730948 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:18.731037 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:18.738445 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:18.738559 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:18.738667 14829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:34:18.748102 14829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:18.749536 14829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:18.751324 14829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:18.764932 14834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:18.765003 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:18.765012 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:18.771337 14829 server_base.cc:1034] running on GCE node
I20260925 01:34:18.772255 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:18.773569 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:18.774820 14829 hybrid_clock.cc:648] HybridClock initialized: now 1790300058774767 us; error 47 us; skew 500 ppm
I20260925 01:34:18.775223 14829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:18.778221 14829 webserver.cc:460] Webserver started at http://127.27.160.2:41279/ using document root <none> and password file <none>
I20260925 01:34:18.779201 14829 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:18.779335 14829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:18.779703 14829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:18.783787 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "26472927f126443e9ab0f203f4913daf"
format_stamp: "Formatted at 2026-09-25 01:34:18 on dist-test-slave-h6hf"
server_key: "8e9da963330465e9db13a1edd63b4323"
server_key_iv: "6d9cd108b78bfad649b6be7964bcfae4"
server_key_version: "encryptionkey@0"
I20260925 01:34:18.784848 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "26472927f126443e9ab0f203f4913daf"
format_stamp: "Formatted at 2026-09-25 01:34:18 on dist-test-slave-h6hf"
server_key: "8e9da963330465e9db13a1edd63b4323"
server_key_iv: "6d9cd108b78bfad649b6be7964bcfae4"
server_key_version: "encryptionkey@0"
I20260925 01:34:18.791821 14829 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260925 01:34:18.796510 14843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:18.798295 14829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:18.798482 14829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "26472927f126443e9ab0f203f4913daf"
format_stamp: "Formatted at 2026-09-25 01:34:18 on dist-test-slave-h6hf"
server_key: "8e9da963330465e9db13a1edd63b4323"
server_key_iv: "6d9cd108b78bfad649b6be7964bcfae4"
server_key_version: "encryptionkey@0"
I20260925 01:34:18.798746 14829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:18.821375 14829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:18.822358 14829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:18.822722 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:18.823822 14829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:18.825415 14829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:18.825503 14829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:18.825649 14829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:18.825731 14829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:18.861580 14829 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:42825
I20260925 01:34:18.861642 14955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:42825 every 8 connection(s)
I20260925 01:34:18.863005 14829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:34:18.864537 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 14829
I20260925 01:34:18.864758 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:34:18.865319 28288 external_mini_cluster.cc:1428] Setting key a4b78349192e4fc3f1398bc7fc116909
I20260925 01:34:18.868639 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41449
--builtin_ntp_servers=127.27.160.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:34:18.877854 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:18.865874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:60117 (local address 127.27.160.62:41449)
0925 01:34:18.866082 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:18.866090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:18.867885 (+ 1795us) server_negotiation.cc:407] Connection header received
0925 01:34:18.870045 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:18.870054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:18.870150 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:18.870271 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:18.871496 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:18.872622 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:18.874515 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:18.875244 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:18.875399 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:18.877004 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:18.877012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:18.877019 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:18.877053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:18.877102 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:18.877165 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:18.877252 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:18.877533 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 01:34:18.877620 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:34:18.879220 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:18.866538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41449 (local address 127.27.160.2:60117)
0925 01:34:18.867689 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:18.867750 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:18.869753 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:18.870509 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:18.870518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:18.870687 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:18.871311 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:18.871331 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:18.872798 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:18.872803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:18.874348 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:18.874362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:18.875488 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:18.875494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:18.875779 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:18.876573 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:18.876576 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:18.876678 (+ 102us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:18.876784 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:18.877275 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:18.877289 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:18.877300 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:34:18.877519 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 01:34:18.878082 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":307,"threads_started":1}
I20260925 01:34:18.880776 14956 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41449
I20260925 01:34:18.881167 14956 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:18.882025 14956 heartbeater.cc:511] Master 127.27.160.62:41449 requested a full tablet report, sending...
I20260925 01:34:18.883596 14639 ts_manager.cc:194] Registered new tserver with Master: 26472927f126443e9ab0f203f4913daf (127.27.160.2:42825)
I20260925 01:34:18.884435 14639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.2:60117
W20260925 01:34:19.227080 14959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:19.227418 14959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:19.227607 14959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:19.236431 14959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:19.236572 14959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:19.236781 14959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:34:19.247531 14959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:41449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:19.249006 14959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:19.251060 14959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:19.266433 14964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:19.266607 14965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:19.266368 14967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:19.268050 14959 server_base.cc:1034] running on GCE node
I20260925 01:34:19.268895 14959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:19.270080 14959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:19.271322 14959 hybrid_clock.cc:648] HybridClock initialized: now 1790300059271263 us; error 46 us; skew 500 ppm
I20260925 01:34:19.271729 14959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:19.278293 14959 webserver.cc:460] Webserver started at http://127.27.160.3:35963/ using document root <none> and password file <none>
I20260925 01:34:19.279137 14959 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:19.279242 14959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:19.279651 14959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:19.282927 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "4a25cedac61741d2814779ce4fa361af"
format_stamp: "Formatted at 2026-09-25 01:34:19 on dist-test-slave-h6hf"
server_key: "2b38794e49c25d859022a37ac02de19e"
server_key_iv: "039b8279d31b1c0e2087c7085e431f92"
server_key_version: "encryptionkey@0"
I20260925 01:34:19.283821 14959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "4a25cedac61741d2814779ce4fa361af"
format_stamp: "Formatted at 2026-09-25 01:34:19 on dist-test-slave-h6hf"
server_key: "2b38794e49c25d859022a37ac02de19e"
server_key_iv: "039b8279d31b1c0e2087c7085e431f92"
server_key_version: "encryptionkey@0"
I20260925 01:34:19.289805 14959 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260925 01:34:19.294150 14973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:19.295809 14959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:34:19.296010 14959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "4a25cedac61741d2814779ce4fa361af"
format_stamp: "Formatted at 2026-09-25 01:34:19 on dist-test-slave-h6hf"
server_key: "2b38794e49c25d859022a37ac02de19e"
server_key_iv: "039b8279d31b1c0e2087c7085e431f92"
server_key_version: "encryptionkey@0"
I20260925 01:34:19.296237 14959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:19.330080 14959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:19.331025 14959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:19.331384 14959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:19.332394 14959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:19.333986 14959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:19.334090 14959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:19.334177 14959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:19.334244 14959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:19.369779 14959 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:46423
I20260925 01:34:19.369903 15085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:46423 every 8 connection(s)
I20260925 01:34:19.371263 14959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:34:19.381044 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 14959
I20260925 01:34:19.381341 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:34:19.381907 28288 external_mini_cluster.cc:1428] Setting key 0112536463e877afba088950ea07cbb4
I20260925 01:34:19.386498 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.374343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:49781 (local address 127.27.160.62:41449)
0925 01:34:19.374602 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.374609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.376402 (+ 1793us) server_negotiation.cc:407] Connection header received
0925 01:34:19.378463 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.378468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.378540 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.378646 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:19.379920 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.380759 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.383182 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.383782 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.383932 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.385701 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:19.385706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:19.385710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:19.385730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:19.385766 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:19.385780 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:19.385879 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.386147 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.386261 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 01:34:19.387562 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.375030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:41449 (local address 127.27.160.3:49781)
0925 01:34:19.376217 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.376264 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:19.378136 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:19.378930 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:19.378938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:19.379034 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:19.379697 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.379725 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.380962 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.380969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.383006 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.383025 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.384059 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.384067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.384352 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.385250 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:19.385253 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:19.385327 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:19.385454 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:19.385894 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:19.385903 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:19.385910 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:34:19.386065 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 01:34:19.386411 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":400,"threads_started":1}
I20260925 01:34:19.389191 15086 heartbeater.cc:347] Connected to a master server at 127.27.160.62:41449
I20260925 01:34:19.389576 15086 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:19.390509 15086 heartbeater.cc:511] Master 127.27.160.62:41449 requested a full tablet report, sending...
I20260925 01:34:19.392355 14639 ts_manager.cc:194] Registered new tserver with Master: 4a25cedac61741d2814779ce4fa361af (127.27.160.3:46423)
I20260925 01:34:19.393407 14639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.3:49781
I20260925 01:34:19.398032 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:34:19.398689 14825 heartbeater.cc:503] Master 127.27.160.62:41449 was elected leader, sending a full tablet report...
I20260925 01:34:19.409453 14827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.402714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.27.160.62:41449)
0925 01:34:19.403003 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.403008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.403504 (+ 496us) server_negotiation.cc:407] Connection header received
0925 01:34:19.403656 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.403662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.403735 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.403833 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:19.404327 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.405239 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.406737 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.407236 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.407355 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.408633 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:19.408638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:19.408642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:19.408663 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:19.408698 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:19.408710 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:19.408838 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.409049 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.409102 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260925 01:34:19.414397 14639 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:34:19.422533 14639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:34:19.449615 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.437713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.27.160.1:40989)
0925 01:34:19.439334 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.439348 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.439394 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:34:19.439534 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.439546 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.439901 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.441045 (+ 1144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:19.441367 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.442922 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.446115 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.447865 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.448027 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.448373 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.448954 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.449200 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":1220,"threads_started":1}
I20260925 01:34:19.449662 15100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.438709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:42825 (local address 127.0.0.1:48024)
0925 01:34:19.439677 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.439699 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:19.439834 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:19.441438 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:19.441447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:19.441484 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:19.441951 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.441967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.444006 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.444014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.446761 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.446776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.448598 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.448605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.448839 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.448884 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 01:34:19.448989 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:34:19.449181 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":312,"thread_start_us":641,"threads_started":1}
I20260925 01:34:19.449702 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.437151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40989 (local address 127.0.0.1:39362)
0925 01:34:19.438105 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.438143 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:19.438304 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:19.440215 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:19.440224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:19.440260 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:19.440805 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.440820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.443096 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.443102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.445875 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.445895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.448123 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.448130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.448367 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.448446 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 01:34:19.448603 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 01:34:19.449155 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":125,"thread_start_us":405,"threads_started":1}
I20260925 01:34:19.450982 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.438963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.27.160.2:42825)
0925 01:34:19.440384 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.440394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.440425 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:34:19.440546 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.440558 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.440847 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.441086 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:19.442179 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.443820 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.446988 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.448362 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.448517 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.448811 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.450362 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.450580 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":2531,"threads_started":1}
I20260925 01:34:19.453438 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.440416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:46423 (local address 127.0.0.1:57278)
0925 01:34:19.442583 (+ 2167us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.442609 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:19.442750 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:19.443947 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:19.443952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:19.443980 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:19.445546 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.445569 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.447304 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.447310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.450903 (+ 3593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.450916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.452549 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.452557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.452751 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.452789 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 01:34:19.452889 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 01:34:19.453041 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2014,"thread_start_us":1880,"threads_started":1}
I20260925 01:34:19.453620 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.440116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.27.160.3:46423)
0925 01:34:19.443378 (+ 3262us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.443385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.443412 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:34:19.443489 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.443494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.443758 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.443969 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:19.445795 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.447118 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.451117 (+ 3999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.452323 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.452488 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.452798 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.452984 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.453239 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3100,"thread_start_us":2939,"threads_started":1}
I20260925 01:34:19.454349 14760 tablet_service.cc:1467] Processing CreateTablet for tablet f14fb6b8344c4e40b56973a76dda2ee6 (DEFAULT_TABLE table=test-table [id=b16e93ed638d493caf29907e712054d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:19.455128 14891 tablet_service.cc:1467] Processing CreateTablet for tablet f14fb6b8344c4e40b56973a76dda2ee6 (DEFAULT_TABLE table=test-table [id=b16e93ed638d493caf29907e712054d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:19.456120 14760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14fb6b8344c4e40b56973a76dda2ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:19.456434 14891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14fb6b8344c4e40b56973a76dda2ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:19.462857 15021 tablet_service.cc:1467] Processing CreateTablet for tablet f14fb6b8344c4e40b56973a76dda2ee6 (DEFAULT_TABLE table=test-table [id=b16e93ed638d493caf29907e712054d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:19.464566 15021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f14fb6b8344c4e40b56973a76dda2ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:19.470727 15104 tablet_bootstrap.cc:492] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: Bootstrap starting.
I20260925 01:34:19.471875 15105 tablet_bootstrap.cc:492] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Bootstrap starting.
I20260925 01:34:19.475135 15104 tablet_bootstrap.cc:654] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:19.475978 15105 tablet_bootstrap.cc:654] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:19.476600 15104 log.cc:824] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:19.477404 15105 log.cc:824] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:19.480728 15104 tablet_bootstrap.cc:492] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: No bootstrap required, opened a new log
I20260925 01:34:19.481246 15104 ts_tablet_manager.cc:1397] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260925 01:34:19.481564 15108 tablet_bootstrap.cc:492] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Bootstrap starting.
I20260925 01:34:19.482918 15105 tablet_bootstrap.cc:492] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: No bootstrap required, opened a new log
I20260925 01:34:19.483292 15105 ts_tablet_manager.cc:1397] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20260925 01:34:19.485806 15108 tablet_bootstrap.cc:654] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:19.487319 15108 log.cc:824] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:19.491086 15104 raft_consensus.cc:348] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.491528 15104 raft_consensus.cc:374] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:19.491639 15104 raft_consensus.cc:729] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0464c45b6b314f429816914ab2967170, State: Initialized, Role: FOLLOWER
I20260925 01:34:19.491931 15108 tablet_bootstrap.cc:492] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: No bootstrap required, opened a new log
I20260925 01:34:19.492331 15108 ts_tablet_manager.cc:1397] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260925 01:34:19.492344 15104 consensus_queue.cc:260] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.494458 15104 ts_tablet_manager.cc:1428] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260925 01:34:19.494963 15105 raft_consensus.cc:348] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.495467 15105 raft_consensus.cc:374] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:19.495597 15105 raft_consensus.cc:729] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26472927f126443e9ab0f203f4913daf, State: Initialized, Role: FOLLOWER
I20260925 01:34:19.496331 15105 consensus_queue.cc:260] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.497609 14956 heartbeater.cc:503] Master 127.27.160.62:41449 was elected leader, sending a full tablet report...
I20260925 01:34:19.498435 15105 ts_tablet_manager.cc:1428] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260925 01:34:19.504487 15108 raft_consensus.cc:348] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.504940 15108 raft_consensus.cc:374] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:19.505082 15108 raft_consensus.cc:729] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a25cedac61741d2814779ce4fa361af, State: Initialized, Role: FOLLOWER
I20260925 01:34:19.505843 15108 consensus_queue.cc:260] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.507099 15086 heartbeater.cc:503] Master 127.27.160.62:41449 was elected leader, sending a full tablet report...
I20260925 01:34:19.507555 15108 ts_tablet_manager.cc:1428] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
W20260925 01:34:19.615259 14957 tablet.cc:2369] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:34:19.623225 15087 tablet.cc:2369] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:34:19.624140 14826 tablet.cc:2369] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:34:19.630313 15110 raft_consensus.cc:482] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:34:19.630695 15110 raft_consensus.cc:504] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.632550 15110 leader_election.cc:288] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26472927f126443e9ab0f203f4913daf (127.27.160.2:42825), 4a25cedac61741d2814779ce4fa361af (127.27.160.3:46423)
I20260925 01:34:19.640453 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.633245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:38227 (local address 127.27.160.2:42825)
0925 01:34:19.633499 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.633506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.633915 (+ 409us) server_negotiation.cc:407] Connection header received
0925 01:34:19.634319 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.634329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.634457 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.634540 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:19.635425 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.636245 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.638499 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.639404 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.639602 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.639690 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.640017 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.640168 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 01:34:19.640492 15113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.632938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:42825 (local address 127.27.160.1:38227)
0925 01:34:19.633759 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.633783 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:19.634067 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:19.634793 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:19.634799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:19.634834 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:19.635254 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.635264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.636388 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.636393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.638265 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.638278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.639665 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.639672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.639835 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.639862 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 01:34:19.639982 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 01:34:19.640131 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":478,"threads_started":1}
I20260925 01:34:19.641685 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.633920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:46423 (local address 127.27.160.1:49289)
0925 01:34:19.634837 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.634854 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:19.635033 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:19.636402 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:19.636406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:19.636431 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:19.636849 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.636858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.638038 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.638044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.639509 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:19.639522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.640902 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.640910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:19.641146 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.641192 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 01:34:19.641310 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 01:34:19.641416 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1,"thread_start_us":565,"threads_started":1}
I20260925 01:34:19.641881 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.634247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:49289 (local address 127.27.160.3:46423)
0925 01:34:19.634448 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.634456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.634954 (+ 498us) server_negotiation.cc:407] Connection header received
0925 01:34:19.635207 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.635212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.635280 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.635391 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:19.637029 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.637877 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.639715 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.640702 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.640865 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.640961 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.641349 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.641504 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:34:19.641834 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14fb6b8344c4e40b56973a76dda2ee6" candidate_uuid: "0464c45b6b314f429816914ab2967170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26472927f126443e9ab0f203f4913daf" is_pre_election: true
I20260925 01:34:19.642563 14911 raft_consensus.cc:2454] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0464c45b6b314f429816914ab2967170 in term 0.
I20260925 01:34:19.643083 15041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14fb6b8344c4e40b56973a76dda2ee6" candidate_uuid: "0464c45b6b314f429816914ab2967170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a25cedac61741d2814779ce4fa361af" is_pre_election: true
I20260925 01:34:19.643409 14713 leader_election.cc:302] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0464c45b6b314f429816914ab2967170, 26472927f126443e9ab0f203f4913daf; no voters:
I20260925 01:34:19.643679 15041 raft_consensus.cc:2454] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0464c45b6b314f429816914ab2967170 in term 0.
I20260925 01:34:19.643925 15110 raft_consensus.cc:2781] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:34:19.644105 15110 raft_consensus.cc:482] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:34:19.644253 15110 raft_consensus.cc:3037] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:19.646991 15110 raft_consensus.cc:504] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.647686 15110 leader_election.cc:288] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [CANDIDATE]: Term 1 election: Requested vote from peers 26472927f126443e9ab0f203f4913daf (127.27.160.2:42825), 4a25cedac61741d2814779ce4fa361af (127.27.160.3:46423)
I20260925 01:34:19.648350 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14fb6b8344c4e40b56973a76dda2ee6" candidate_uuid: "0464c45b6b314f429816914ab2967170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26472927f126443e9ab0f203f4913daf"
I20260925 01:34:19.648379 15041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f14fb6b8344c4e40b56973a76dda2ee6" candidate_uuid: "0464c45b6b314f429816914ab2967170" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a25cedac61741d2814779ce4fa361af"
I20260925 01:34:19.648602 14911 raft_consensus.cc:3037] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:19.648701 15041 raft_consensus.cc:3037] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:19.651234 15041 raft_consensus.cc:2454] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0464c45b6b314f429816914ab2967170 in term 1.
I20260925 01:34:19.651268 14911 raft_consensus.cc:2454] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0464c45b6b314f429816914ab2967170 in term 1.
I20260925 01:34:19.651768 14713 leader_election.cc:302] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0464c45b6b314f429816914ab2967170, 4a25cedac61741d2814779ce4fa361af; no voters:
I20260925 01:34:19.652175 15110 raft_consensus.cc:2781] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:19.652709 15110 raft_consensus.cc:686] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 1 LEADER]: Becoming Leader. State: Replica: 0464c45b6b314f429816914ab2967170, State: Running, Role: LEADER
I20260925 01:34:19.653431 15110 consensus_queue.cc:237] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } }
I20260925 01:34:19.658914 14638 catalog_manager.cc:5495] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0464c45b6b314f429816914ab2967170 (127.27.160.1). New cstate: current_term: 1 leader_uuid: "0464c45b6b314f429816914ab2967170" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0464c45b6b314f429816914ab2967170" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 40989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 } health_report { overall_health: UNKNOWN } } }
I20260925 01:34:19.695624 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.689752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.27.160.1:40989)
0925 01:34:19.689950 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.689955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.689973 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:34:19.690080 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.690084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.690224 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.690371 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:34:19.690733 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.691546 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.693239 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.693726 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.693847 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.694399 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:34:19.694955 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:34:19.695103 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.695256 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.695332 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:34:19.705660 15041 raft_consensus.cc:1261] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 1 FOLLOWER]: Refusing update from remote peer 0464c45b6b314f429816914ab2967170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:19.705725 14911 raft_consensus.cc:1261] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 1 FOLLOWER]: Refusing update from remote peer 0464c45b6b314f429816914ab2967170: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:19.706782 15115 consensus_queue.cc:1035] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a25cedac61741d2814779ce4fa361af" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:19.707238 15110 consensus_queue.cc:1035] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26472927f126443e9ab0f203f4913daf" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 42825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:19.740047 14639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:19.740424 14639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:19.744732 14639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 045600befc7443b0a7b73f7a197a1f03: Sending DeleteTablet for 3 replicas of tablet f14fb6b8344c4e40b56973a76dda2ee6
I20260925 01:34:19.746142 14760 tablet_service.cc:1514] Processing DeleteTablet for tablet f14fb6b8344c4e40b56973a76dda2ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:19 UTC) from {username='slave'} at 127.0.0.1:39362
I20260925 01:34:19.746312 15021 tablet_service.cc:1514] Processing DeleteTablet for tablet f14fb6b8344c4e40b56973a76dda2ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:19 UTC) from {username='slave'} at 127.0.0.1:57278
I20260925 01:34:19.746968 14891 tablet_service.cc:1514] Processing DeleteTablet for tablet f14fb6b8344c4e40b56973a76dda2ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:19 UTC) from {username='slave'} at 127.0.0.1:48024
I20260925 01:34:19.748011 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 14698
I20260925 01:34:19.758458 15129 tablet_replica.cc:330] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: stopping tablet replica
I20260925 01:34:19.758950 15129 raft_consensus.cc:2229] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:19.759495 15129 raft_consensus.cc:2258] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:19.762707 15129 ts_tablet_manager.cc:1905] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:19.764005 15130 tablet_replica.cc:330] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: stopping tablet replica
I20260925 01:34:19.764488 15130 raft_consensus.cc:2229] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:19.765054 15130 raft_consensus.cc:2258] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:19.767529 15130 ts_tablet_manager.cc:1905] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:19.771863 15129 ts_tablet_manager.cc:1918] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:19.772183 15129 log.cc:1196] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/f14fb6b8344c4e40b56973a76dda2ee6
I20260925 01:34:19.772897 15129 ts_tablet_manager.cc:1939] T f14fb6b8344c4e40b56973a76dda2ee6 P 4a25cedac61741d2814779ce4fa361af: Deleting consensus metadata
I20260925 01:34:19.775620 14623 catalog_manager.cc:4841] TS 4a25cedac61741d2814779ce4fa361af (127.27.160.3:46423): tablet f14fb6b8344c4e40b56973a76dda2ee6 (table test-table [id=b16e93ed638d493caf29907e712054d1]) successfully deleted
I20260925 01:34:19.775866 15128 tablet_replica.cc:330] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: stopping tablet replica
I20260925 01:34:19.776180 15130 ts_tablet_manager.cc:1918] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:19.776403 15130 log.cc:1196] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/f14fb6b8344c4e40b56973a76dda2ee6
I20260925 01:34:19.776403 15128 raft_consensus.cc:2229] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:34:19.776925 15130 ts_tablet_manager.cc:1939] T f14fb6b8344c4e40b56973a76dda2ee6 P 26472927f126443e9ab0f203f4913daf: Deleting consensus metadata
I20260925 01:34:19.777216 15128 raft_consensus.cc:2258] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:19.778906 14623 catalog_manager.cc:4841] TS 26472927f126443e9ab0f203f4913daf (127.27.160.2:42825): tablet f14fb6b8344c4e40b56973a76dda2ee6 (table test-table [id=b16e93ed638d493caf29907e712054d1]) successfully deleted
I20260925 01:34:19.783547 15128 ts_tablet_manager.cc:1905] T f14fb6b8344c4e40b56973a76dda2ee6 P 0464c45b6b314f429816914ab2967170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:19.783829 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.750897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.27.160.1:40989)
0925 01:34:19.751135 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.751143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.764498 (+ 13355us) server_negotiation.cc:407] Connection header received
0925 01:34:19.765335 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.765342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.765482 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.765571 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:19.766869 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.768263 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.772792 (+ 4529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.773654 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.773772 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.781809 (+ 8037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:19.781821 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:19.781840 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:19.782763 (+ 923us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:19.782840 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:19.782871 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:19.782996 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.783264 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.783387 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:34:19.785539 14820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:19.895874 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 14698
I20260925 01:34:19.916730 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 14829
W20260925 01:34:19.916730 14623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.160.1:40989 (error 108)
I20260925 01:34:19.918912 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.918250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40989 (local address 127.0.0.1:39372)
0925 01:34:19.918452 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.918711 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 01:34:19.919214 14623 catalog_manager.cc:4584] TS 0464c45b6b314f429816914ab2967170 (127.27.160.1:40989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f14fb6b8344c4e40b56973a76dda2ee6: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40989: connect: Connection refused (error 111)
I20260925 01:34:19.924827 15101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.917762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.27.160.2:42825)
0925 01:34:19.917941 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:19.917946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:19.917994 (+ 48us) server_negotiation.cc:407] Connection header received
0925 01:34:19.918250 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:19.918256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:19.918387 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:19.918506 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:19.919042 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.919940 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.921604 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:19.922104 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:19.922203 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:19.923357 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:19.923365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:19.923379 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:19.923971 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:19.924017 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:19.924036 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:19.924141 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:19.924327 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 01:34:19.924414 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:34:19.925720 14951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:19.973369 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.972725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40989 (local address 127.0.0.1:39388)
0925 01:34:19.972959 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.973069 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 01:34:19.974538 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:19.974183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40989 (local address 127.0.0.1:39392)
0925 01:34:19.974337 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:19.974417 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 01:34:20.019055 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 14829
I20260925 01:34:20.035261 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:20.034521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40989 (local address 127.0.0.1:39406)
0925 01:34:20.034833 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:20.034982 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 01:34:20.036556 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:20.036027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40989 (local address 127.0.0.1:39422)
0925 01:34:20.036251 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:20.036361 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 01:34:20.038342 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 14959
I20260925 01:34:20.045506 15102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:20.039119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.27.160.3:46423)
0925 01:34:20.039301 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:20.039306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:20.039325 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:34:20.039512 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:20.039516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:20.039610 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:20.039718 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:20.040082 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:20.040830 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:20.042327 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:20.042746 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:20.042847 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:20.043911 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:20.043921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:20.043938 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:20.044648 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:20.044698 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:20.044718 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:20.044811 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:20.044943 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 01:34:20.045033 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:34:20.046284 15081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:20.117969 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:20.117383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40989 (local address 127.0.0.1:39434)
0925 01:34:20.117622 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:20.117750 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 01:34:20.119240 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:20.118750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:40989 (local address 127.0.0.1:39446)
0925 01:34:20.118984 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:20.119079 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260925 01:34:20.147902 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 14959
I20260925 01:34:20.165158 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 14608
I20260925 01:34:20.166388 14669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:20.283746 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 14608
2026-09-25T01:34:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3033 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:20 dist-test-slave-h6hf krb5kdc[15139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:20 dist-test-slave-h6hf krb5kdc[15139](info): set up 2 sockets
Sep 25 01:34:20 dist-test-slave-h6hf krb5kdc[15139](info): commencing operation
krb5kdc: starting...
W20260925 01:34:21.434834 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:21 dist-test-slave-h6hf krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:34:21.711095 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:34327
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:22.044981 15152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:22.045463 15152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:22.045634 15152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:22.053089 15152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:22.053328 15152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:22.053442 15152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:22.053550 15152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:22.053622 15152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:22.063905 15152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:22.065364 15152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:22.066725 15152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:22.079455 15157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:22.079594 15158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:22.079558 15160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:22.080672 15152 server_base.cc:1034] running on GCE node
I20260925 01:34:22.081566 15152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:22.083045 15152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:22.084301 15152 hybrid_clock.cc:648] HybridClock initialized: now 1790300062084276 us; error 81 us; skew 500 ppm
I20260925 01:34:22.084671 15152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:22 dist-test-slave-h6hf krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300062, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:22.088007 15152 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:34:22.089720 15152 webserver.cc:460] Webserver started at http://127.27.160.62:38527/ using document root <none> and password file <none>
I20260925 01:34:22.090425 15152 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:22.090576 15152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:22.090895 15152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:22.094051 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "be50ff325ba44a6f822c977d838614db"
format_stamp: "Formatted at 2026-09-25 01:34:22 on dist-test-slave-h6hf"
server_key: "a696c5011844191f9131e5a56b5069c7"
server_key_iv: "a8224cfd981c4023bfaa88d7d2afc5b3"
server_key_version: "encryptionkey@0"
I20260925 01:34:22.094827 15152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "be50ff325ba44a6f822c977d838614db"
format_stamp: "Formatted at 2026-09-25 01:34:22 on dist-test-slave-h6hf"
server_key: "a696c5011844191f9131e5a56b5069c7"
server_key_iv: "a8224cfd981c4023bfaa88d7d2afc5b3"
server_key_version: "encryptionkey@0"
I20260925 01:34:22.100077 15152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:34:22.103765 15167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:22.105199 15152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:22.105388 15152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "be50ff325ba44a6f822c977d838614db"
format_stamp: "Formatted at 2026-09-25 01:34:22 on dist-test-slave-h6hf"
server_key: "a696c5011844191f9131e5a56b5069c7"
server_key_iv: "a8224cfd981c4023bfaa88d7d2afc5b3"
server_key_version: "encryptionkey@0"
I20260925 01:34:22.105600 15152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:22.117208 15152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:22.121703 15152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:22.122014 15152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:22.141526 15152 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:34327
I20260925 01:34:22.141548 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:34327 every 8 connection(s)
I20260925 01:34:22.143057 15152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:22.146873 15219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:22.147087 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 15152
I20260925 01:34:22.147282 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:22.147746 28288 external_mini_cluster.cc:1428] Setting key 8cbcef2b326e3335bb1bcf8f417a43ed
I20260925 01:34:22.159471 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db: Bootstrap starting.
I20260925 01:34:22.162627 15219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db: Neither blocks nor log segments found. Creating new log.
Sep 25 01:34:22 dist-test-slave-h6hf krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:22.163710 15219 log.cc:824] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:22.166814 15219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db: No bootstrap required, opened a new log
I20260925 01:34:22.172565 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:22.149552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.27.160.62:34327)
0925 01:34:22.150395 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:22.150406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:22.150447 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:34:22.152938 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:22.152963 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:22.153534 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:22.153964 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:22.155622 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:22.157070 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:22.158596 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:22.159346 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:22.159570 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:22.164233 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:22.164242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:22.164258 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:22.164305 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:22.167883 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:22.168662 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:22.168677 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:22.168683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:22.168873 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:22.169401 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:22.169414 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:22.169417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:22.169993 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:22.170362 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:22.170923 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 01:34:22.171426 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":381,"threads_started":1}
I20260925 01:34:22.175155 15219 raft_consensus.cc:348] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50ff325ba44a6f822c977d838614db" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 34327 } }
I20260925 01:34:22.175571 15219 raft_consensus.cc:374] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:22.175702 15219 raft_consensus.cc:729] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be50ff325ba44a6f822c977d838614db, State: Initialized, Role: FOLLOWER
I20260925 01:34:22.176316 15219 consensus_queue.cc:260] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50ff325ba44a6f822c977d838614db" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 34327 } }
I20260925 01:34:22.176571 15219 raft_consensus.cc:388] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:22.176712 15219 raft_consensus.cc:482] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:22.176891 15219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:22.179301 15219 raft_consensus.cc:504] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50ff325ba44a6f822c977d838614db" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 34327 } }
I20260925 01:34:22.179823 15219 leader_election.cc:302] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be50ff325ba44a6f822c977d838614db; no voters:
I20260925 01:34:22.180428 15219 leader_election.cc:288] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:22.180610 15224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:22.181481 15224 raft_consensus.cc:686] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [term 1 LEADER]: Becoming Leader. State: Replica: be50ff325ba44a6f822c977d838614db, State: Running, Role: LEADER
I20260925 01:34:22.182090 15224 consensus_queue.cc:237] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50ff325ba44a6f822c977d838614db" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 34327 } }
I20260925 01:34:22.182768 15219 sys_catalog.cc:564] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:22.185091 15225 sys_catalog.cc:455] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be50ff325ba44a6f822c977d838614db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50ff325ba44a6f822c977d838614db" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 34327 } } }
I20260925 01:34:22.185379 15226 sys_catalog.cc:455] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [sys.catalog]: SysCatalogTable state changed. Reason: New leader be50ff325ba44a6f822c977d838614db. Latest consensus state: current_term: 1 leader_uuid: "be50ff325ba44a6f822c977d838614db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be50ff325ba44a6f822c977d838614db" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 34327 } } }
I20260925 01:34:22.185693 15226 sys_catalog.cc:458] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:22.185595 15225 sys_catalog.cc:458] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:22.186697 15229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:22.192585 15229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:22.208154 15229 catalog_manager.cc:1347] Generated new cluster ID: bd03d7c456b2458e9d0430b419f9a537
I20260925 01:34:22.208293 15229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:22.224807 15229 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:22.226291 15229 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:22.232997 15229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db: Generated new TSK 0
I20260925 01:34:22.234159 15229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.160.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
Entry for principal kudu/127.27.160.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab.
I20260925 01:34:22.310953 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:34327
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:22.662043 15245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:22.662364 15245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:22.662560 15245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:22.670172 15245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:22.670306 15245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:22.670575 15245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:34:22.681443 15245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.1.keytab
--principal=kudu/127.27.160.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:22.682926 15245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:22.684744 15245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:22.699626 15250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:22.699700 15253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:22.699760 15251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:22.700924 15245 server_base.cc:1034] running on GCE node
I20260925 01:34:22.701696 15245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:22.702793 15245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:22.704054 15245 hybrid_clock.cc:648] HybridClock initialized: now 1790300062704007 us; error 38 us; skew 500 ppm
I20260925 01:34:22.704476 15245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:22 dist-test-slave-h6hf krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300062, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:22.708307 15245 init.cc:377] Logged in from keytab as kudu/127.27.160.1@KRBTEST.COM (short username kudu)
I20260925 01:34:22.710347 15245 webserver.cc:460] Webserver started at http://127.27.160.1:37059/ using document root <none> and password file <none>
I20260925 01:34:22.711191 15245 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:22.711360 15245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:22.711735 15245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:22.715134 15245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "19fd207865694466b7ef4eb80e40a25b"
format_stamp: "Formatted at 2026-09-25 01:34:22 on dist-test-slave-h6hf"
server_key: "f65d98338fd0b07d306bf8005e17e355"
server_key_iv: "0dde121e1e9905b67729cf2bd438d44a"
server_key_version: "encryptionkey@0"
I20260925 01:34:22.716043 15245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "19fd207865694466b7ef4eb80e40a25b"
format_stamp: "Formatted at 2026-09-25 01:34:22 on dist-test-slave-h6hf"
server_key: "f65d98338fd0b07d306bf8005e17e355"
server_key_iv: "0dde121e1e9905b67729cf2bd438d44a"
server_key_version: "encryptionkey@0"
I20260925 01:34:22.722275 15245 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 01:34:22.726470 15260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:22.728117 15245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:34:22.728341 15245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "19fd207865694466b7ef4eb80e40a25b"
format_stamp: "Formatted at 2026-09-25 01:34:22 on dist-test-slave-h6hf"
server_key: "f65d98338fd0b07d306bf8005e17e355"
server_key_iv: "0dde121e1e9905b67729cf2bd438d44a"
server_key_version: "encryptionkey@0"
I20260925 01:34:22.728593 15245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:22.758905 15245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:22.763626 15245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:22.763945 15245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:22.764950 15245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:22.766363 15245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:22.766464 15245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:22.766608 15245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:22.766680 15245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:22.801396 15245 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:38557
I20260925 01:34:22.801433 15372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:38557 every 8 connection(s)
I20260925 01:34:22.802723 15245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:34:22.811492 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 15245
I20260925 01:34:22.811770 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:34:22.812287 28288 external_mini_cluster.cc:1428] Setting key dc77b219a5fa9a571a41d22a743dc97f
Sep 25 01:34:22 dist-test-slave-h6hf krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300062, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.1@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:22.823051 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:22.805352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:60267 (local address 127.27.160.62:34327)
0925 01:34:22.805811 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:22.805816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:22.806772 (+ 956us) server_negotiation.cc:407] Connection header received
0925 01:34:22.808710 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:22.808716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:22.808811 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:22.808940 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:22.810750 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:22.811505 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:22.813259 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:22.813694 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:22.813828 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:22.817293 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:22.817299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:22.817302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:22.817335 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:22.819906 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:22.820656 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:22.820666 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:22.820668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:22.820787 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:22.821280 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:22.821285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:22.821288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:22.821570 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:22.821729 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:22.822448 (+ 719us) server_negotiation.cc:299] Negotiation successful
0925 01:34:22.822699 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":263,"threads_started":1}
I20260925 01:34:22.824087 15376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:22.805856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:34327 (local address 127.27.160.1:60267)
0925 01:34:22.806550 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:22.806641 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:22.808408 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:22.809252 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:22.809292 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:22.809928 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:22.810580 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:22.810599 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:22.811675 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:22.811680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:22.813045 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:22.813062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:22.813941 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:22.813950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:22.814195 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:22.814898 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:22.814922 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:22.817034 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:22.820062 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:22.820071 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:22.820089 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:22.820515 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:22.820900 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:22.820905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:22.820908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:22.821110 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:22.821700 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:22.821708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:22.822058 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 01:34:22.822328 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 01:34:22.822896 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":271,"threads_started":1}
I20260925 01:34:22.825760 15373 heartbeater.cc:347] Connected to a master server at 127.27.160.62:34327
I20260925 01:34:22.826083 15373 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:22.826792 15373 heartbeater.cc:511] Master 127.27.160.62:34327 requested a full tablet report, sending...
I20260925 01:34:22.829218 15184 ts_manager.cc:194] Registered new tserver with Master: 19fd207865694466b7ef4eb80e40a25b (127.27.160.1:38557)
I20260925 01:34:22.831446 15184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.1@KRBTEST.COM'} at 127.27.160.1:60267
WARNING: no policy specified for kudu/127.27.160.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
Entry for principal kudu/127.27.160.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab.
I20260925 01:34:22.873631 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:34327
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:23.224723 15379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:23.225054 15379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:23.225240 15379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:23.232865 15379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:23.233019 15379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:23.233238 15379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:34:23.243290 15379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.2.keytab
--principal=kudu/127.27.160.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:23.244705 15379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:23.246131 15379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:23.259207 15384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:23.259168 15385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:23.259315 15387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:23.260346 15379 server_base.cc:1034] running on GCE node
I20260925 01:34:23.261020 15379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:23.262022 15379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:23.263309 15379 hybrid_clock.cc:648] HybridClock initialized: now 1790300063263252 us; error 48 us; skew 500 ppm
I20260925 01:34:23.263664 15379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:23 dist-test-slave-h6hf krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:23.267172 15379 init.cc:377] Logged in from keytab as kudu/127.27.160.2@KRBTEST.COM (short username kudu)
I20260925 01:34:23.269030 15379 webserver.cc:460] Webserver started at http://127.27.160.2:39915/ using document root <none> and password file <none>
I20260925 01:34:23.269876 15379 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:23.270051 15379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:23.270457 15379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:23.273952 15379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "60889eb6cf584892b83eff4243e6f6c2"
format_stamp: "Formatted at 2026-09-25 01:34:23 on dist-test-slave-h6hf"
server_key: "ee746398f57c215785557aee5519af1d"
server_key_iv: "c84926cdd3239241c6b848997b228579"
server_key_version: "encryptionkey@0"
I20260925 01:34:23.274827 15379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "60889eb6cf584892b83eff4243e6f6c2"
format_stamp: "Formatted at 2026-09-25 01:34:23 on dist-test-slave-h6hf"
server_key: "ee746398f57c215785557aee5519af1d"
server_key_iv: "c84926cdd3239241c6b848997b228579"
server_key_version: "encryptionkey@0"
I20260925 01:34:23.280870 15379 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 01:34:23.285252 15394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:23.286798 15379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:23.287012 15379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "60889eb6cf584892b83eff4243e6f6c2"
format_stamp: "Formatted at 2026-09-25 01:34:23 on dist-test-slave-h6hf"
server_key: "ee746398f57c215785557aee5519af1d"
server_key_iv: "c84926cdd3239241c6b848997b228579"
server_key_version: "encryptionkey@0"
I20260925 01:34:23.287245 15379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:23.304731 15379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:23.309429 15379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:23.309814 15379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:23.310850 15379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:23.312269 15379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:23.312368 15379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:23.312512 15379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:23.312587 15379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:23.350276 15379 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:35897
I20260925 01:34:23.350343 15506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:35897 every 8 connection(s)
I20260925 01:34:23.351732 15379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:34:23.352813 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 15379
I20260925 01:34:23.353016 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:34:23.353735 28288 external_mini_cluster.cc:1428] Setting key c45e49b2df560b7daf7f50c47f338537
Sep 25 01:34:23 dist-test-slave-h6hf krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.2@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:23.375819 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.354867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:58505 (local address 127.27.160.62:34327)
0925 01:34:23.355427 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:23.355434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:23.357054 (+ 1620us) server_negotiation.cc:407] Connection header received
0925 01:34:23.358911 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:23.358917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:23.359026 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:23.359156 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:23.360774 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.361669 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.363438 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.363960 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.364118 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:23.369068 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:23.369073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:23.369077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:23.369110 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:23.371912 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:23.372973 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:23.372977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:23.372979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:23.373110 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:23.373748 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:23.373752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:23.373754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:23.374069 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:23.374197 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:23.375254 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0925 01:34:23.375447 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
I20260925 01:34:23.377434 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.355596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:34327 (local address 127.27.160.2:58505)
0925 01:34:23.356514 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:23.356553 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:23.358588 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:23.359423 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:23.359432 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:23.359976 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:23.360588 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:23.360609 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.361840 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.361844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:23.363276 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:23.363289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.364216 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.364225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:23.364486 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:23.365343 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:23.365379 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:23.368786 (+ 3407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:23.372076 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:23.372088 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:23.372111 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:23.372840 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:23.373303 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:23.373312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:23.373319 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:23.373605 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:23.374230 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:23.374241 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:23.374788 (+ 547us) client_negotiation.cc:769] Sending connection context
0925 01:34:23.375120 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 01:34:23.375785 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":403,"threads_started":1}
I20260925 01:34:23.379232 15507 heartbeater.cc:347] Connected to a master server at 127.27.160.62:34327
I20260925 01:34:23.379640 15507 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:23.380722 15507 heartbeater.cc:511] Master 127.27.160.62:34327 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.160.3@KRBTEST.COM; defaulting to no policy
I20260925 01:34:23.382797 15184 ts_manager.cc:194] Registered new tserver with Master: 60889eb6cf584892b83eff4243e6f6c2 (127.27.160.2:35897)
I20260925 01:34:23.383912 15184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.2@KRBTEST.COM'} at 127.27.160.2:58505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
Entry for principal kudu/127.27.160.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab.
I20260925 01:34:23.417676 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:34327
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:23.772516 15513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:23.772899 15513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:23.773000 15513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:23.780978 15513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:23.781162 15513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:23.781306 15513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:34:23.791821 15513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.3.keytab
--principal=kudu/127.27.160.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:23.793385 15513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:23.794741 15513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:23.810171 15519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:23.810849 15521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:23.810894 15518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:23.811069 15513 server_base.cc:1034] running on GCE node
I20260925 01:34:23.811821 15513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:23.812907 15513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:23.814169 15513 hybrid_clock.cc:648] HybridClock initialized: now 1790300063814079 us; error 93 us; skew 500 ppm
I20260925 01:34:23.814515 15513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:23 dist-test-slave-h6hf krb5kdc[15139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:23.817836 15513 init.cc:377] Logged in from keytab as kudu/127.27.160.3@KRBTEST.COM (short username kudu)
I20260925 01:34:23.819370 15513 webserver.cc:460] Webserver started at http://127.27.160.3:46523/ using document root <none> and password file <none>
I20260925 01:34:23.820048 15513 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:23.820159 15513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:23.820425 15513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:23.823568 15513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "0a5725ac8f0e45a9b20800292ff60bd8"
format_stamp: "Formatted at 2026-09-25 01:34:23 on dist-test-slave-h6hf"
server_key: "41465749376d0fcf798c15200e6bc6e7"
server_key_iv: "2eea1db1b4881ab1a4b9401501a851e3"
server_key_version: "encryptionkey@0"
I20260925 01:34:23.824363 15513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "0a5725ac8f0e45a9b20800292ff60bd8"
format_stamp: "Formatted at 2026-09-25 01:34:23 on dist-test-slave-h6hf"
server_key: "41465749376d0fcf798c15200e6bc6e7"
server_key_iv: "2eea1db1b4881ab1a4b9401501a851e3"
server_key_version: "encryptionkey@0"
I20260925 01:34:23.829831 15513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:34:23.833657 15528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:23.834985 15373 heartbeater.cc:503] Master 127.27.160.62:34327 was elected leader, sending a full tablet report...
I20260925 01:34:23.835321 15513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:23.835511 15513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "0a5725ac8f0e45a9b20800292ff60bd8"
format_stamp: "Formatted at 2026-09-25 01:34:23 on dist-test-slave-h6hf"
server_key: "41465749376d0fcf798c15200e6bc6e7"
server_key_iv: "2eea1db1b4881ab1a4b9401501a851e3"
server_key_version: "encryptionkey@0"
I20260925 01:34:23.835744 15513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:23.859477 15513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:23.864454 15513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:23.864820 15513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:23.865932 15513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:23.867344 15513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:23.867461 15513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:23.867626 15513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:23.867702 15513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:23.911504 15513 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:33669
I20260925 01:34:23.911542 15640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:33669 every 8 connection(s)
I20260925 01:34:23.913527 15513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:34:23.921348 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 15513
I20260925 01:34:23.921568 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:34:23.922199 28288 external_mini_cluster.cc:1428] Setting key 6b6c7d631d4725e553a63f0a2441eccd
Sep 25 01:34:23 dist-test-slave-h6hf krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300063, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.3@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:23.937719 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.917883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:60831 (local address 127.27.160.62:34327)
0925 01:34:23.918358 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:23.918364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:23.918388 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:34:23.920276 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:23.920282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:23.920369 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:23.920497 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:23.922452 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.923517 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.925827 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.926594 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.926787 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:23.930648 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:23.930657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:23.930673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:23.930724 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:23.933806 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:23.934718 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:23.934723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:23.934726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:23.934882 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:23.935377 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:23.935387 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:23.935389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:23.935759 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:23.935911 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:23.936969 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0925 01:34:23.937290 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":219,"threads_started":1}
I20260925 01:34:23.938877 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.917229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:34327 (local address 127.27.160.3:60831)
0925 01:34:23.918214 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:23.918277 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:23.919958 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:23.920746 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:23.920754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:23.921459 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:23.922281 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:23.922306 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.923804 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.923808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:23.925667 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:23.925683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.926858 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.926863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:23.927061 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:23.927777 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:34:23.927800 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:23.930373 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:23.933950 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:23.933957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:23.933972 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:23.934491 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:23.935011 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:34:23.935015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:34:23.935017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:34:23.935230 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:34:23.935921 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:23.935929 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:23.936447 (+ 518us) client_negotiation.cc:769] Sending connection context
0925 01:34:23.936815 (+ 368us) client_negotiation.cc:240] Negotiation successful
0925 01:34:23.937454 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":368,"threads_started":1}
I20260925 01:34:23.940515 15641 heartbeater.cc:347] Connected to a master server at 127.27.160.62:34327
I20260925 01:34:23.940904 15641 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:23.941736 15641 heartbeater.cc:511] Master 127.27.160.62:34327 requested a full tablet report, sending...
I20260925 01:34:23.943269 15184 ts_manager.cc:194] Registered new tserver with Master: 0a5725ac8f0e45a9b20800292ff60bd8 (127.27.160.3:33669)
I20260925 01:34:23.944059 15184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.160.3@KRBTEST.COM'} at 127.27.160.3:60831
I20260925 01:34:23.951483 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:34:23.970201 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.955944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42212 (local address 127.27.160.62:34327)
0925 01:34:23.956112 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:23.956116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:23.956522 (+ 406us) server_negotiation.cc:407] Connection header received
0925 01:34:23.956682 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:23.956686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:23.956765 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:23.956853 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:23.958525 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.959441 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.960791 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:23.961320 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:23.961429 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:23.963931 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:23.963936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:23.963940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:23.963979 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:23.967250 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:23.968012 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:23.968019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:23.968023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:23.968170 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:23.968578 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:23.968586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:23.968589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:23.968932 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:23.969248 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:23.969619 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 01:34:23.969872 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:34:23.974635 15184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:34:23.981679 15184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:34:24.022346 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.994544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:35897 (local address 127.0.0.1:52462)
0925 01:34:24.007005 (+ 12461us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:24.007070 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:24.007346 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:24.010898 (+ 3552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:24.010909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:24.010955 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:24.011494 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.011511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.014435 (+ 2924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.014444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.017826 (+ 3382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.017847 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.021355 (+ 3508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.021361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.021585 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.021662 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 01:34:24.021773 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:34:24.021976 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12043,"thread_start_us":15075,"threads_started":1}
I20260925 01:34:24.023634 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.994703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.27.160.2:35897)
0925 01:34:24.009984 (+ 15281us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.009994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.010025 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:34:24.010126 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.010134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.010397 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.011794 (+ 1397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:24.012077 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.014234 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.018125 (+ 3891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.020225 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.022782 (+ 2557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.023018 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.023083 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.023301 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15085,"thread_start_us":15839,"threads_started":1}
I20260925 01:34:24.024035 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.995061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:33669 (local address 127.0.0.1:59156)
0925 01:34:24.009538 (+ 14477us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:24.009562 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:24.009722 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:24.012537 (+ 2815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:24.012545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:24.012580 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:24.013134 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.013148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.017444 (+ 4296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.017452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.021345 (+ 3893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.021357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.023242 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.023248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.023429 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.023465 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:34:24.023547 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 01:34:24.023694 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14299,"thread_start_us":12562,"threads_started":1}
I20260925 01:34:24.024308 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:23.995523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.27.160.3:33669)
0925 01:34:24.009772 (+ 14249us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.009781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.009819 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:34:24.009939 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.009949 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.010395 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.010651 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:24.013499 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.017225 (+ 3726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.021516 (+ 4291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.023019 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.023170 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.023394 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.023783 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.023947 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14063,"thread_start_us":13847,"threads_started":1}
I20260925 01:34:24.028100 15442 tablet_service.cc:1467] Processing CreateTablet for tablet 884de1b75d354eb7a9e4a076dec85b05 (DEFAULT_TABLE table=test-table [id=8e79d27405c74b6d84e88d99d286cc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:24.028100 15576 tablet_service.cc:1467] Processing CreateTablet for tablet 884de1b75d354eb7a9e4a076dec85b05 (DEFAULT_TABLE table=test-table [id=8e79d27405c74b6d84e88d99d286cc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:24.028944 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.010078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:38557 (local address 127.0.0.1:55138)
0925 01:34:24.020946 (+ 10868us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:24.020976 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:24.021136 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:24.022323 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:24.022332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:24.022358 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:24.022764 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.022777 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.024765 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.024770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.026705 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.026719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.028113 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.028119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.028290 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.028324 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 01:34:24.028451 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 01:34:24.028588 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10674,"thread_start_us":10439,"threads_started":1}
I20260925 01:34:24.029392 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.010515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.27.160.1:38557)
0925 01:34:24.021791 (+ 11276us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.021798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.021822 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:34:24.021901 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.021907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.022122 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.022345 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:24.022961 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.024562 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.026886 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.027899 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.028066 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.028321 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.028493 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.028648 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11086,"thread_start_us":10936,"threads_started":1}
I20260925 01:34:24.029677 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884de1b75d354eb7a9e4a076dec85b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:24.029696 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884de1b75d354eb7a9e4a076dec85b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:24.032964 15308 tablet_service.cc:1467] Processing CreateTablet for tablet 884de1b75d354eb7a9e4a076dec85b05 (DEFAULT_TABLE table=test-table [id=8e79d27405c74b6d84e88d99d286cc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:24.034307 15308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 884de1b75d354eb7a9e4a076dec85b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:24.045972 15660 tablet_bootstrap.cc:492] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Bootstrap starting.
I20260925 01:34:24.046268 15661 tablet_bootstrap.cc:492] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Bootstrap starting.
I20260925 01:34:24.047426 15662 tablet_bootstrap.cc:492] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Bootstrap starting.
I20260925 01:34:24.050523 15662 tablet_bootstrap.cc:654] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:24.050529 15660 tablet_bootstrap.cc:654] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:24.051824 15662 log.cc:824] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:24.051873 15660 log.cc:824] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:24.053544 15661 tablet_bootstrap.cc:654] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:24.054726 15661 log.cc:824] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:24.055193 15660 tablet_bootstrap.cc:492] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: No bootstrap required, opened a new log
I20260925 01:34:24.055315 15662 tablet_bootstrap.cc:492] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: No bootstrap required, opened a new log
I20260925 01:34:24.055527 15660 ts_tablet_manager.cc:1397] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 01:34:24.055627 15662 ts_tablet_manager.cc:1397] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260925 01:34:24.058133 15661 tablet_bootstrap.cc:492] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: No bootstrap required, opened a new log
I20260925 01:34:24.058471 15661 ts_tablet_manager.cc:1397] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260925 01:34:24.063320 15660 raft_consensus.cc:348] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.063701 15660 raft_consensus.cc:374] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:24.063799 15660 raft_consensus.cc:729] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60889eb6cf584892b83eff4243e6f6c2, State: Initialized, Role: FOLLOWER
I20260925 01:34:24.064419 15660 consensus_queue.cc:260] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.065575 15507 heartbeater.cc:503] Master 127.27.160.62:34327 was elected leader, sending a full tablet report...
I20260925 01:34:24.066293 15660 ts_tablet_manager.cc:1428] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 01:34:24.066236 15662 raft_consensus.cc:348] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.066632 15662 raft_consensus.cc:374] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:24.066733 15662 raft_consensus.cc:729] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fd207865694466b7ef4eb80e40a25b, State: Initialized, Role: FOLLOWER
I20260925 01:34:24.067344 15662 consensus_queue.cc:260] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.069099 15662 ts_tablet_manager.cc:1428] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Time spent starting tablet: real 0.013s user 0.002s sys 0.010s
I20260925 01:34:24.069300 15661 raft_consensus.cc:348] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.069723 15661 raft_consensus.cc:374] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:24.069823 15661 raft_consensus.cc:729] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5725ac8f0e45a9b20800292ff60bd8, State: Initialized, Role: FOLLOWER
I20260925 01:34:24.070407 15661 consensus_queue.cc:260] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.072824 15641 heartbeater.cc:503] Master 127.27.160.62:34327 was elected leader, sending a full tablet report...
I20260925 01:34:24.073576 15661 ts_tablet_manager.cc:1428] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260925 01:34:24.104154 15508 tablet.cc:2369] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:34:24.165362 15642 tablet.cc:2369] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:34:24.305974 15374 tablet.cc:2369] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:34:24.350627 15668 raft_consensus.cc:482] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:34:24.351039 15668 raft_consensus.cc:504] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.353250 15668 leader_election.cc:288] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60889eb6cf584892b83eff4243e6f6c2 (127.27.160.2:35897), 19fd207865694466b7ef4eb80e40a25b (127.27.160.1:38557)
I20260925 01:34:24.361311 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.353843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:35897 (local address 127.27.160.3:41783)
0925 01:34:24.354044 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:24.354070 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:24.354240 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:24.355712 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:24.355718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:24.355758 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:24.355955 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.355967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.357603 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.357609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.359172 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.359186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.360462 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.360470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.360686 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.360713 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 01:34:24.360826 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:34:24.360982 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 01:34:24.361635 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.353795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:41783 (local address 127.27.160.2:35897)
0925 01:34:24.354862 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.354873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.354896 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:34:24.354973 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.354980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.355079 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.355176 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:24.356123 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.357433 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.359347 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.360288 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.360488 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.360572 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.360918 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.361072 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878}
I20260925 01:34:24.362905 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.354347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:38557 (local address 127.27.160.3:35897)
0925 01:34:24.354999 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:24.355018 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:24.355176 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:24.356282 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:24.356285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:24.356308 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:24.356622 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.356629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.357973 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.357979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.360301 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:24.360318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.362253 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.362260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:24.362408 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.362432 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:34:24.362500 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 01:34:24.362606 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":1,"thread_start_us":1708,"threads_started":1}
I20260925 01:34:24.362725 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884de1b75d354eb7a9e4a076dec85b05" candidate_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60889eb6cf584892b83eff4243e6f6c2" is_pre_election: true
I20260925 01:34:24.363366 15462 raft_consensus.cc:2454] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a5725ac8f0e45a9b20800292ff60bd8 in term 0.
I20260925 01:34:24.363785 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.355526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:35897 (local address 127.27.160.1:38557)
0925 01:34:24.355760 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.355766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.355785 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:34:24.355876 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.355884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.355964 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.356088 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:24.356834 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.357789 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.360508 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.362028 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.363286 (+ 1258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.363352 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.363398 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.363492 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 01:34:24.364235 15531 leader_election.cc:302] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5725ac8f0e45a9b20800292ff60bd8, 60889eb6cf584892b83eff4243e6f6c2; no voters:
I20260925 01:34:24.364738 15668 raft_consensus.cc:2781] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:34:24.364902 15668 raft_consensus.cc:482] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:34:24.365042 15668 raft_consensus.cc:3037] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:24.364820 15328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884de1b75d354eb7a9e4a076dec85b05" candidate_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fd207865694466b7ef4eb80e40a25b" is_pre_election: true
I20260925 01:34:24.365429 15328 raft_consensus.cc:2454] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a5725ac8f0e45a9b20800292ff60bd8 in term 0.
I20260925 01:34:24.367821 15668 raft_consensus.cc:504] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.368587 15668 leader_election.cc:288] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [CANDIDATE]: Term 1 election: Requested vote from peers 60889eb6cf584892b83eff4243e6f6c2 (127.27.160.2:35897), 19fd207865694466b7ef4eb80e40a25b (127.27.160.1:38557)
I20260925 01:34:24.369097 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884de1b75d354eb7a9e4a076dec85b05" candidate_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60889eb6cf584892b83eff4243e6f6c2"
I20260925 01:34:24.369154 15328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "884de1b75d354eb7a9e4a076dec85b05" candidate_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fd207865694466b7ef4eb80e40a25b"
I20260925 01:34:24.369364 15462 raft_consensus.cc:3037] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:24.369359 15328 raft_consensus.cc:3037] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:24.372968 15462 raft_consensus.cc:2454] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a5725ac8f0e45a9b20800292ff60bd8 in term 1.
I20260925 01:34:24.372968 15328 raft_consensus.cc:2454] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a5725ac8f0e45a9b20800292ff60bd8 in term 1.
I20260925 01:34:24.373497 15531 leader_election.cc:302] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5725ac8f0e45a9b20800292ff60bd8, 60889eb6cf584892b83eff4243e6f6c2; no voters:
I20260925 01:34:24.373855 15668 raft_consensus.cc:2781] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:24.374387 15668 raft_consensus.cc:686] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 0a5725ac8f0e45a9b20800292ff60bd8, State: Running, Role: LEADER
I20260925 01:34:24.375007 15668 consensus_queue.cc:237] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } }
I20260925 01:34:24.379580 15183 catalog_manager.cc:5495] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a5725ac8f0e45a9b20800292ff60bd8 (127.27.160.3). New cstate: current_term: 1 leader_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a5725ac8f0e45a9b20800292ff60bd8" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 33669 } health_report { overall_health: HEALTHY } } }
I20260925 01:34:24.476119 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.470251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.27.160.3:33669)
0925 01:34:24.470504 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.470510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.471026 (+ 516us) server_negotiation.cc:407] Connection header received
0925 01:34:24.471201 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.471205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.471302 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.471432 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:34:24.471828 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.472513 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.474017 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.474401 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.474494 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.474976 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:34:24.475514 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:34:24.475626 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.475741 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.475811 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:34:24.484639 15462 raft_consensus.cc:1261] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 1 FOLLOWER]: Refusing update from remote peer 0a5725ac8f0e45a9b20800292ff60bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:24.484652 15328 raft_consensus.cc:1261] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 1 FOLLOWER]: Refusing update from remote peer 0a5725ac8f0e45a9b20800292ff60bd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:24.485656 15670 consensus_queue.cc:1035] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60889eb6cf584892b83eff4243e6f6c2" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 35897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:24.486032 15668 consensus_queue.cc:1035] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fd207865694466b7ef4eb80e40a25b" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 38557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:24.515450 15184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:24.515805 15184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:24.520258 15184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be50ff325ba44a6f822c977d838614db: Sending DeleteTablet for 3 replicas of tablet 884de1b75d354eb7a9e4a076dec85b05
I20260925 01:34:24.521502 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 884de1b75d354eb7a9e4a076dec85b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:24 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:52462
I20260925 01:34:24.521552 15576 tablet_service.cc:1514] Processing DeleteTablet for tablet 884de1b75d354eb7a9e4a076dec85b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:24 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:59156
I20260925 01:34:24.521548 15308 tablet_service.cc:1514] Processing DeleteTablet for tablet 884de1b75d354eb7a9e4a076dec85b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:24 UTC) from {username='kudu', principal='kudu/127.27.160.62@KRBTEST.COM'} at 127.0.0.1:55138
I20260925 01:34:24.524168 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 15245
I20260925 01:34:24.537703 15685 tablet_replica.cc:330] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: stopping tablet replica
I20260925 01:34:24.539342 15686 tablet_replica.cc:330] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: stopping tablet replica
I20260925 01:34:24.539901 15686 raft_consensus.cc:2229] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:34:24.538236 15685 raft_consensus.cc:2229] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:24.541630 15685 raft_consensus.cc:2258] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:24.542337 15686 raft_consensus.cc:2258] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:24.543649 15684 tablet_replica.cc:330] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: stopping tablet replica
I20260925 01:34:24.544152 15684 raft_consensus.cc:2229] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:24.544775 15684 raft_consensus.cc:2258] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:24.544956 15686 ts_tablet_manager.cc:1905] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:24.546895 15684 ts_tablet_manager.cc:1905] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:24.546895 15685 ts_tablet_manager.cc:1905] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:24.554414 15684 ts_tablet_manager.cc:1918] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:24.554617 15684 log.cc:1196] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/884de1b75d354eb7a9e4a076dec85b05
I20260925 01:34:24.554414 15685 ts_tablet_manager.cc:1918] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:24.555238 15684 ts_tablet_manager.cc:1939] T 884de1b75d354eb7a9e4a076dec85b05 P 60889eb6cf584892b83eff4243e6f6c2: Deleting consensus metadata
I20260925 01:34:24.555241 15685 log.cc:1196] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/884de1b75d354eb7a9e4a076dec85b05
I20260925 01:34:24.555896 15685 ts_tablet_manager.cc:1939] T 884de1b75d354eb7a9e4a076dec85b05 P 19fd207865694466b7ef4eb80e40a25b: Deleting consensus metadata
I20260925 01:34:24.558058 15170 catalog_manager.cc:4841] TS 19fd207865694466b7ef4eb80e40a25b (127.27.160.1:38557): tablet 884de1b75d354eb7a9e4a076dec85b05 (table test-table [id=8e79d27405c74b6d84e88d99d286cc6d]) successfully deleted
I20260925 01:34:24.558475 15170 catalog_manager.cc:4841] TS 60889eb6cf584892b83eff4243e6f6c2 (127.27.160.2:35897): tablet 884de1b75d354eb7a9e4a076dec85b05 (table test-table [id=8e79d27405c74b6d84e88d99d286cc6d]) successfully deleted
I20260925 01:34:24.558772 15686 ts_tablet_manager.cc:1918] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:24.558975 15686 log.cc:1196] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/884de1b75d354eb7a9e4a076dec85b05
I20260925 01:34:24.559577 15686 ts_tablet_manager.cc:1939] T 884de1b75d354eb7a9e4a076dec85b05 P 0a5725ac8f0e45a9b20800292ff60bd8: Deleting consensus metadata
Sep 25 01:34:24 dist-test-slave-h6hf krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.1@KRBTEST.COM
I20260925 01:34:24.564654 15168 catalog_manager.cc:4841] TS 0a5725ac8f0e45a9b20800292ff60bd8 (127.27.160.3:33669): tablet 884de1b75d354eb7a9e4a076dec85b05 (table test-table [id=8e79d27405c74b6d84e88d99d286cc6d]) successfully deleted
I20260925 01:34:24.576167 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.525867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.27.160.1:38557)
0925 01:34:24.526062 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.526068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.531007 (+ 4939us) server_negotiation.cc:407] Connection header received
0925 01:34:24.531944 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.531952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.532067 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.532148 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:24.536477 (+ 4329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.537830 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.552609 (+ 14779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.553471 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.553598 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.566807 (+ 13209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:24.566835 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:24.566857 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:24.566939 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:24.571005 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:24.572083 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:24.572101 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:24.572108 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:24.572324 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:24.572913 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:24.572922 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:24.572927 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:24.573532 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:24.573896 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.575403 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.575707 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:34:24.577313 15368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:24.671789 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 15245
I20260925 01:34:24.690596 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 15379
Sep 25 01:34:24 dist-test-slave-h6hf krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.2@KRBTEST.COM
I20260925 01:34:24.709831 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.691455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.27.160.2:35897)
0925 01:34:24.691656 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.691663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.691714 (+ 51us) server_negotiation.cc:407] Connection header received
0925 01:34:24.691899 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.691905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.692004 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.692126 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:24.694115 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.695243 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.696692 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.697404 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.697541 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.702528 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:24.702552 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:24.702574 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:24.702623 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:24.706558 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:24.707369 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:24.707379 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:24.707383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:24.707546 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:24.707949 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:24.707954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:24.707957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:24.708453 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:24.708786 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.709217 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.709452 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:34:24.710521 15502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:24.807107 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 15379
I20260925 01:34:24.831430 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 15513
Sep 25 01:34:24 dist-test-slave-h6hf krb5kdc[15139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.3@KRBTEST.COM
I20260925 01:34:24.849226 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:24.832476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.27.160.3:33669)
0925 01:34:24.832754 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:24.832760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:24.832781 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:34:24.832859 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:24.832863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:24.832922 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:24.833039 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:24.834465 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.835316 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.836785 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:24.837392 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:24.837579 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:24.842522 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:24.842538 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:24.842561 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:24.842611 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:24.846034 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:24.846835 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:24.846843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:24.846846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:24.847002 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:24.847431 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:24.847438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:24.847445 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:24.847832 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:24.848196 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:24.848536 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 01:34:24.848801 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260925 01:34:24.850005 15636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:24.981491 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 15513
I20260925 01:34:25.002027 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 15152
I20260925 01:34:25.003333 15214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:25.117612 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 15152
2026-09-25T01:34:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4834 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T01:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:25Z Disabled control of system clock
I20260925 01:34:25.199090 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:33983
--webserver_interface=127.27.160.62
--webserver_port=0
--builtin_ntp_servers=127.27.160.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:33983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:34:25.551764 15695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:25.552081 15695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:25.552235 15695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:25.559777 15695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:25.559896 15695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:25.559967 15695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:25.560061 15695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:25.560137 15695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:25.569736 15695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:33983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:25.571151 15695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:25.573596 15695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:25.585820 15700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:25.585785 15701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:25.585919 15703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:25.586771 15695 server_base.cc:1034] running on GCE node
I20260925 01:34:25.587577 15695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:25.589056 15695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:25.590363 15695 hybrid_clock.cc:648] HybridClock initialized: now 1790300065590276 us; error 60 us; skew 500 ppm
I20260925 01:34:25.590752 15695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:25.593039 15695 webserver.cc:460] Webserver started at http://127.27.160.62:46587/ using document root <none> and password file <none>
I20260925 01:34:25.593873 15695 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:25.594013 15695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:25.594340 15695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:25.597353 15695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "9d8cce2438334cde8358610c7363d189"
format_stamp: "Formatted at 2026-09-25 01:34:25 on dist-test-slave-h6hf"
server_key: "47a3b46a2457ded5477719b11ef115d6"
server_key_iv: "0c7ba4838da49f5adac301b58c352993"
server_key_version: "encryptionkey@0"
I20260925 01:34:25.598186 15695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "9d8cce2438334cde8358610c7363d189"
format_stamp: "Formatted at 2026-09-25 01:34:25 on dist-test-slave-h6hf"
server_key: "47a3b46a2457ded5477719b11ef115d6"
server_key_iv: "0c7ba4838da49f5adac301b58c352993"
server_key_version: "encryptionkey@0"
I20260925 01:34:25.603394 15695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 01:34:25.607085 15709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:25.608639 15695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:34:25.608822 15695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "9d8cce2438334cde8358610c7363d189"
format_stamp: "Formatted at 2026-09-25 01:34:25 on dist-test-slave-h6hf"
server_key: "47a3b46a2457ded5477719b11ef115d6"
server_key_iv: "0c7ba4838da49f5adac301b58c352993"
server_key_version: "encryptionkey@0"
I20260925 01:34:25.609047 15695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:25.629426 15695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:25.630300 15695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:25.630599 15695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:25.650861 15695 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:33983
I20260925 01:34:25.650880 15760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:33983 every 8 connection(s)
I20260925 01:34:25.652308 15695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:25.656488 15761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:25.657552 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 15695
I20260925 01:34:25.657747 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:25.658322 28288 external_mini_cluster.cc:1428] Setting key 6d899e400e7df4ff6d5d339b34db3ffc
I20260925 01:34:25.671108 15761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189: Bootstrap starting.
I20260925 01:34:25.673331 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:25.660367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.27.160.62:33983)
0925 01:34:25.661527 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:25.661545 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:25.661590 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:34:25.663232 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:25.663254 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:25.663752 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:25.664095 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:25.664733 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:25.666090 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:25.668117 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:25.669085 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:25.669347 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:25.670644 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:25.670652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:25.670669 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:25.671304 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:25.671390 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:25.671411 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:25.671501 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:25.671820 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 01:34:25.672197 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":422,"threads_started":1}
I20260925 01:34:25.674613 15761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:25.675673 15761 log.cc:824] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:25.678560 15761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189: No bootstrap required, opened a new log
I20260925 01:34:25.688534 15761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8cce2438334cde8358610c7363d189" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } }
I20260925 01:34:25.689021 15761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:25.689172 15761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8cce2438334cde8358610c7363d189, State: Initialized, Role: FOLLOWER
I20260925 01:34:25.689849 15761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8cce2438334cde8358610c7363d189" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } }
I20260925 01:34:25.690099 15761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:25.690235 15761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:25.690400 15761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:25.692741 15761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8cce2438334cde8358610c7363d189" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } }
I20260925 01:34:25.693356 15761 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d8cce2438334cde8358610c7363d189; no voters:
I20260925 01:34:25.694101 15761 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:25.694296 15766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:25.695186 15766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [term 1 LEADER]: Becoming Leader. State: Replica: 9d8cce2438334cde8358610c7363d189, State: Running, Role: LEADER
I20260925 01:34:25.695760 15766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8cce2438334cde8358610c7363d189" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } }
I20260925 01:34:25.696686 15761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:25.699036 15768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8cce2438334cde8358610c7363d189. Latest consensus state: current_term: 1 leader_uuid: "9d8cce2438334cde8358610c7363d189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8cce2438334cde8358610c7363d189" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } }
I20260925 01:34:25.699501 15768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:25.699548 15767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d8cce2438334cde8358610c7363d189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d8cce2438334cde8358610c7363d189" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 33983 } } }
I20260925 01:34:25.699966 15767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:25.700842 15774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:25.706589 15774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:25.715626 15774 catalog_manager.cc:1347] Generated new cluster ID: 4f58adc11f6b4ce4ac27e87d73f71e12
I20260925 01:34:25.715759 15774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:25.748194 15774 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:25.750176 15774 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:25.763446 15774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189: Generated new TSK 0
I20260925 01:34:25.764588 15774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:25.776302 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.1:0
--local_ip_for_outbound_sockets=127.27.160.1
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--builtin_ntp_servers=127.27.160.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:34:26.123800 15785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:26.124089 15785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:26.124269 15785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:26.131703 15785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:26.131824 15785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:26.132005 15785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.1
I20260925 01:34:26.141443 15785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.160.1
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.1
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:26.142788 15785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:26.144578 15785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:26.157806 15791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:26.157729 15790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:26.157755 15793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:26.158679 15785 server_base.cc:1034] running on GCE node
I20260925 01:34:26.159368 15785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:26.160410 15785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:26.161628 15785 hybrid_clock.cc:648] HybridClock initialized: now 1790300066161579 us; error 39 us; skew 500 ppm
I20260925 01:34:26.161912 15785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:26.164204 15785 webserver.cc:460] Webserver started at http://127.27.160.1:36273/ using document root <none> and password file <none>
I20260925 01:34:26.164883 15785 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:26.165009 15785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:26.165359 15785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:26.168118 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/data/instance:
uuid: "540b9975c27d42f7b0f8422a6c3f58ed"
format_stamp: "Formatted at 2026-09-25 01:34:26 on dist-test-slave-h6hf"
server_key: "99b025cbd3f3e1bb87ac6f7299a05338"
server_key_iv: "a976ccf68f746cc28bec8ec4892e9d19"
server_key_version: "encryptionkey@0"
I20260925 01:34:26.168861 15785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance:
uuid: "540b9975c27d42f7b0f8422a6c3f58ed"
format_stamp: "Formatted at 2026-09-25 01:34:26 on dist-test-slave-h6hf"
server_key: "99b025cbd3f3e1bb87ac6f7299a05338"
server_key_iv: "a976ccf68f746cc28bec8ec4892e9d19"
server_key_version: "encryptionkey@0"
I20260925 01:34:26.174445 15785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 01:34:26.177949 15799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:26.179364 15785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:26.179508 15785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/wal
uuid: "540b9975c27d42f7b0f8422a6c3f58ed"
format_stamp: "Formatted at 2026-09-25 01:34:26 on dist-test-slave-h6hf"
server_key: "99b025cbd3f3e1bb87ac6f7299a05338"
server_key_iv: "a976ccf68f746cc28bec8ec4892e9d19"
server_key_version: "encryptionkey@0"
I20260925 01:34:26.179751 15785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:26.220454 15785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:26.221362 15785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:26.221632 15785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:26.222589 15785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:26.224009 15785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:26.224097 15785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:26.224169 15785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:26.224226 15785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:26.257977 15785 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.1:46557
I20260925 01:34:26.258025 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.1:46557 every 8 connection(s)
I20260925 01:34:26.259315 15785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/data/info.pb
I20260925 01:34:26.266332 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 15785
I20260925 01:34:26.266522 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/wal/instance
I20260925 01:34:26.266956 28288 external_mini_cluster.cc:1428] Setting key b39a0fe1f9d9cb91ad864558b38a7912
I20260925 01:34:26.269891 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.2:0
--local_ip_for_outbound_sockets=127.27.160.2
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--builtin_ntp_servers=127.27.160.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:34:26.273218 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:26.262150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.1:54793 (local address 127.27.160.62:33983)
0925 01:34:26.262625 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:26.262629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:26.263921 (+ 1292us) server_negotiation.cc:407] Connection header received
0925 01:34:26.265825 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:26.265831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:26.265947 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:26.266080 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:26.267353 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:26.268217 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:26.270134 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:26.270561 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:26.270685 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:26.272285 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:26.272290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:26.272293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:26.272313 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:26.272354 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:26.272366 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:26.272470 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:26.272718 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 01:34:26.272822 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260925 01:34:26.274327 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:26.262795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.1:54793)
0925 01:34:26.263757 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:26.263801 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:26.265528 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:26.266313 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:26.266322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:26.266492 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:26.267176 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:26.267197 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:26.268394 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:26.268398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:26.269919 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:26.269933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:26.270956 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:26.270961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:26.271187 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:26.271940 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:26.271942 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:26.271985 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:26.272073 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:26.272469 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:26.272483 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:26.272494 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:34:26.272693 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 01:34:26.273126 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":302,"threads_started":1}
I20260925 01:34:26.276170 15912 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:34:26.276584 15912 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:26.277650 15912 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
I20260925 01:34:26.280083 15726 ts_manager.cc:194] Registered new tserver with Master: 540b9975c27d42f7b0f8422a6c3f58ed (127.27.160.1:46557)
I20260925 01:34:26.281754 15726 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.1:54793
W20260925 01:34:26.612335 15916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:26.612623 15916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:26.612769 15916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:26.620152 15916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:26.620253 15916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:26.620419 15916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.2
I20260925 01:34:26.629938 15916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.160.2
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.2
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:26.631268 15916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:26.633088 15916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:26.647207 15924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:26.647359 15921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:26.647743 15922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:26.648663 15916 server_base.cc:1034] running on GCE node
I20260925 01:34:26.649453 15916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:26.650542 15916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:26.651793 15916 hybrid_clock.cc:648] HybridClock initialized: now 1790300066651750 us; error 54 us; skew 500 ppm
I20260925 01:34:26.652168 15916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:26.654676 15916 webserver.cc:460] Webserver started at http://127.27.160.2:43707/ using document root <none> and password file <none>
I20260925 01:34:26.655421 15916 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:26.655512 15916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:26.655861 15916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:26.658999 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/data/instance:
uuid: "cc73f57058024f0689710bb22763dbae"
format_stamp: "Formatted at 2026-09-25 01:34:26 on dist-test-slave-h6hf"
server_key: "e49fad5dabc780a6602fa092b3629a05"
server_key_iv: "a56f908503df7bf01fb91e10cce23e95"
server_key_version: "encryptionkey@0"
I20260925 01:34:26.659894 15916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance:
uuid: "cc73f57058024f0689710bb22763dbae"
format_stamp: "Formatted at 2026-09-25 01:34:26 on dist-test-slave-h6hf"
server_key: "e49fad5dabc780a6602fa092b3629a05"
server_key_iv: "a56f908503df7bf01fb91e10cce23e95"
server_key_version: "encryptionkey@0"
I20260925 01:34:26.665555 15916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 01:34:26.669469 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:26.671167 15916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:26.671401 15916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/wal
uuid: "cc73f57058024f0689710bb22763dbae"
format_stamp: "Formatted at 2026-09-25 01:34:26 on dist-test-slave-h6hf"
server_key: "e49fad5dabc780a6602fa092b3629a05"
server_key_iv: "a56f908503df7bf01fb91e10cce23e95"
server_key_version: "encryptionkey@0"
I20260925 01:34:26.671655 15916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:26.690290 15916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:26.691200 15916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:26.691532 15916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:26.692606 15916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:26.694224 15916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:26.694343 15916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:26.694478 15916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:26.694577 15916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:26.729836 15916 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.2:39803
I20260925 01:34:26.729898 16042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.2:39803 every 8 connection(s)
I20260925 01:34:26.731274 15916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/data/info.pb
I20260925 01:34:26.739679 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 15916
I20260925 01:34:26.739890 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/wal/instance
I20260925 01:34:26.740403 28288 external_mini_cluster.cc:1428] Setting key ceb5877781edaa8c4a058ab89948b02f
I20260925 01:34:26.743448 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.160.3:0
--local_ip_for_outbound_sockets=127.27.160.3
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--builtin_ntp_servers=127.27.160.20:44797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:34:26.746121 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:26.734194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.2:45283 (local address 127.27.160.62:33983)
0925 01:34:26.734416 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:26.734423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:26.736162 (+ 1739us) server_negotiation.cc:407] Connection header received
0925 01:34:26.738218 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:26.738223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:26.738298 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:26.738398 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:26.739902 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:26.740773 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:26.742810 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:26.743264 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:26.743387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:26.745281 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:26.745287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:26.745290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:26.745311 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:26.745351 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:26.745365 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:26.745461 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:26.745754 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 01:34:26.745846 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:34:26.747640 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:26.734875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.2:45283)
0925 01:34:26.735977 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:26.736018 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:26.737903 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:26.738711 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:26.738721 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:26.738823 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:26.739650 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:26.739693 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:26.740934 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:26.740940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:26.742636 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:26.742653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:26.743519 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:26.743528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:26.743895 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:26.744812 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:26.744819 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:26.744878 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:26.744993 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:26.745487 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:26.745500 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:26.745511 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:34:26.745725 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 01:34:26.746193 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":348,"threads_started":1}
I20260925 01:34:26.749233 16043 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:34:26.749557 16043 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:26.750334 16043 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
I20260925 01:34:26.751888 15726 ts_manager.cc:194] Registered new tserver with Master: cc73f57058024f0689710bb22763dbae (127.27.160.2:39803)
I20260925 01:34:26.752950 15726 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.2:45283
W20260925 01:34:27.086467 16046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:27.086773 16046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:27.086860 16046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:27.094368 16046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:27.094487 16046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:27.094607 16046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.160.3
I20260925 01:34:27.104099 16046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.160.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.160.3
--webserver_port=0
--tserver_master_addrs=127.27.160.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.160.3
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:27.105438 16046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:27.107391 16046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:27.121567 16051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:27.121799 16054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:27.121587 16052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:27.122973 16046 server_base.cc:1034] running on GCE node
I20260925 01:34:27.123628 16046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:27.124678 16046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:27.125918 16046 hybrid_clock.cc:648] HybridClock initialized: now 1790300067125828 us; error 73 us; skew 500 ppm
I20260925 01:34:27.126379 16046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:34:27.129249 16046 webserver.cc:460] Webserver started at http://127.27.160.3:46089/ using document root <none> and password file <none>
I20260925 01:34:27.130069 16046 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:27.130187 16046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:27.130586 16046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:27.134168 16046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/data/instance:
uuid: "0935a5422dc74024bbe9e6f7e5330b4c"
format_stamp: "Formatted at 2026-09-25 01:34:27 on dist-test-slave-h6hf"
server_key: "3238e33890d569f0130d2958eca189d5"
server_key_iv: "62e7deadae17de0b7e8a4d20794c74dc"
server_key_version: "encryptionkey@0"
I20260925 01:34:27.135120 16046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance:
uuid: "0935a5422dc74024bbe9e6f7e5330b4c"
format_stamp: "Formatted at 2026-09-25 01:34:27 on dist-test-slave-h6hf"
server_key: "3238e33890d569f0130d2958eca189d5"
server_key_iv: "62e7deadae17de0b7e8a4d20794c74dc"
server_key_version: "encryptionkey@0"
I20260925 01:34:27.141623 16046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 01:34:27.145905 16060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:27.147571 16046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:34:27.147809 16046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/wal
uuid: "0935a5422dc74024bbe9e6f7e5330b4c"
format_stamp: "Formatted at 2026-09-25 01:34:27 on dist-test-slave-h6hf"
server_key: "3238e33890d569f0130d2958eca189d5"
server_key_iv: "62e7deadae17de0b7e8a4d20794c74dc"
server_key_version: "encryptionkey@0"
I20260925 01:34:27.148069 16046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:27.165334 16046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:27.166280 16046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:27.166543 16046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:27.167485 16046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:34:27.169018 16046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:34:27.169104 16046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:27.169261 16046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:34:27.169322 16046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:27.212208 16046 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.3:46201
I20260925 01:34:27.212278 16172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.3:46201 every 8 connection(s)
I20260925 01:34:27.214129 16046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/data/info.pb
I20260925 01:34:27.223269 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 16046
I20260925 01:34:27.223466 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/wal/instance
I20260925 01:34:27.224081 28288 external_mini_cluster.cc:1428] Setting key 1812c912baff43da39270372c68ba3ff
I20260925 01:34:27.229553 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.216967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:50855 (local address 127.27.160.62:33983)
0925 01:34:27.217264 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.217270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.219007 (+ 1737us) server_negotiation.cc:407] Connection header received
0925 01:34:27.220961 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.220966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.221043 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.221244 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:27.222712 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.223686 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.226283 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.226825 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.227005 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.228543 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:27.228548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:27.228551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:27.228571 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:27.228604 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:27.228616 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:27.228791 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.229049 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.229187 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:34:27.230667 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.217734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:33983 (local address 127.27.160.3:50855)
0925 01:34:27.218794 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.218836 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:27.220612 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:27.221519 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:27.221531 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:27.221700 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:34:27.222531 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.222549 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.223850 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.223856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.226046 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.226060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.227098 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.227107 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.227406 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.228165 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:34:27.228168 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:34:27.228221 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:34:27.228312 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:34:27.228801 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:34:27.228815 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:34:27.228826 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:34:27.229037 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 01:34:27.229524 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":341,"threads_started":1}
I20260925 01:34:27.232141 16173 heartbeater.cc:347] Connected to a master server at 127.27.160.62:33983
I20260925 01:34:27.232515 16173 heartbeater.cc:464] Registering TS with master...
I20260925 01:34:27.233475 16173 heartbeater.cc:511] Master 127.27.160.62:33983 requested a full tablet report, sending...
I20260925 01:34:27.235001 15726 ts_manager.cc:194] Registered new tserver with Master: 0935a5422dc74024bbe9e6f7e5330b4c (127.27.160.3:46201)
I20260925 01:34:27.235908 15726 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.160.3:50855
I20260925 01:34:27.240278 28288 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:34:27.252584 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.245494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.27.160.62:33983)
0925 01:34:27.245861 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.245871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.246241 (+ 370us) server_negotiation.cc:407] Connection header received
0925 01:34:27.246401 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.246405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.246480 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.246624 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:27.247193 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.248124 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.249854 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.250356 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.250495 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.251743 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:27.251748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:27.251752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:27.251774 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:27.251835 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:27.251855 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:27.251938 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.252099 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.252191 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 01:34:27.256992 15726 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:34:27.264997 15726 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:34:27.285533 15912 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
I20260925 01:34:27.300675 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.279346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.27.160.1:46557)
0925 01:34:27.289891 (+ 10545us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.289904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.289945 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:34:27.290085 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.290099 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.290519 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.290823 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:27.292166 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.293754 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.297368 (+ 3614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.298934 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.299094 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.299410 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.300126 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.300308 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10330,"thread_start_us":10169,"threads_started":1}
I20260925 01:34:27.301765 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.279085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48182)
0925 01:34:27.288350 (+ 9265us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.288397 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:27.288631 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:27.291273 (+ 2642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:27.291287 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:27.291341 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:27.291928 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.291943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.293916 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.293923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.297112 (+ 3189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.297148 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.299540 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.299546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.299824 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.299892 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 01:34:27.300981 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0925 01:34:27.301390 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8897,"mutex_wait_us":188,"thread_start_us":9762,"threads_started":1}
I20260925 01:34:27.304026 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.280324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:39803 (local address 127.0.0.1:55988)
0925 01:34:27.289076 (+ 8752us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.289096 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:27.289252 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:27.296072 (+ 6820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:27.296085 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:27.296118 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:27.296738 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.296758 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.298748 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.298753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.300853 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.300868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.303232 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.303239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.303401 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.303431 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 01:34:27.303519 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 01:34:27.303658 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8572,"mutex_wait_us":70,"thread_start_us":7446,"threads_started":1}
I20260925 01:34:27.304160 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.279708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.27.160.2:39803)
0925 01:34:27.295331 (+ 15623us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.295342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.295377 (+ 35us) server_negotiation.cc:407] Connection header received
0925 01:34:27.295499 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.295510 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.295840 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.296081 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:27.296957 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.298506 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.301097 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.303015 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.303207 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.303518 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.303612 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.303792 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15446,"thread_start_us":15313,"threads_started":1}
I20260925 01:34:27.306164 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.290335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.3:46201 (local address 127.0.0.1:51226)
0925 01:34:27.294618 (+ 4283us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.294641 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:27.294799 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:27.296302 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:27.296312 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:27.296333 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:27.297090 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.297107 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.300557 (+ 3450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.300563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.302487 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.302500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.305359 (+ 2859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.305367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.305537 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.305562 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:34:27.305673 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:34:27.305820 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4131,"mutex_wait_us":69,"thread_start_us":3833,"threads_started":1}
I20260925 01:34:27.306262 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.289731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.27.160.3:46201)
0925 01:34:27.292248 (+ 2517us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.292263 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.295645 (+ 3382us) server_negotiation.cc:407] Connection header received
0925 01:34:27.295781 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.295789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.296072 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.296651 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:27.298779 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.300395 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.302696 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.304801 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.304945 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.305273 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.305750 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.305928 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2254,"thread_start_us":2065,"threads_started":1}
I20260925 01:34:27.308768 15847 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6986495ad490899294215a83a99ff (DEFAULT_TABLE table=test-table [id=b0f5c72f374d4ccd99dacab414d34c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:27.308719 15978 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6986495ad490899294215a83a99ff (DEFAULT_TABLE table=test-table [id=b0f5c72f374d4ccd99dacab414d34c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:27.310377 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6986495ad490899294215a83a99ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:27.310424 15847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6986495ad490899294215a83a99ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:27.310385 16108 tablet_service.cc:1467] Processing CreateTablet for tablet b9d6986495ad490899294215a83a99ff (DEFAULT_TABLE table=test-table [id=b0f5c72f374d4ccd99dacab414d34c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:34:27.311863 16108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d6986495ad490899294215a83a99ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:27.325896 16191 tablet_bootstrap.cc:492] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Bootstrap starting.
I20260925 01:34:27.328296 16192 tablet_bootstrap.cc:492] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Bootstrap starting.
I20260925 01:34:27.330204 16191 tablet_bootstrap.cc:654] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:27.331635 16191 log.cc:824] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:27.332453 16192 tablet_bootstrap.cc:654] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:27.333928 16192 log.cc:824] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:27.334136 16193 tablet_bootstrap.cc:492] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Bootstrap starting.
I20260925 01:34:27.335713 16191 tablet_bootstrap.cc:492] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: No bootstrap required, opened a new log
I20260925 01:34:27.336073 16191 ts_tablet_manager.cc:1397] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 01:34:27.338238 16192 tablet_bootstrap.cc:492] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: No bootstrap required, opened a new log
I20260925 01:34:27.338222 16193 tablet_bootstrap.cc:654] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:27.338536 16192 ts_tablet_manager.cc:1397] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 01:34:27.339581 16193 log.cc:824] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:27.343406 16193 tablet_bootstrap.cc:492] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: No bootstrap required, opened a new log
I20260925 01:34:27.343741 16193 ts_tablet_manager.cc:1397] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 01:34:27.345309 16191 raft_consensus.cc:348] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.345747 16191 raft_consensus.cc:374] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:27.345883 16191 raft_consensus.cc:729] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540b9975c27d42f7b0f8422a6c3f58ed, State: Initialized, Role: FOLLOWER
I20260925 01:34:27.346580 16191 consensus_queue.cc:260] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.350775 16191 ts_tablet_manager.cc:1428] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260925 01:34:27.350718 16192 raft_consensus.cc:348] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.351245 16192 raft_consensus.cc:374] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:27.351362 16192 raft_consensus.cc:729] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc73f57058024f0689710bb22763dbae, State: Initialized, Role: FOLLOWER
I20260925 01:34:27.352207 16192 consensus_queue.cc:260] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.353682 16043 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
I20260925 01:34:27.354633 16192 ts_tablet_manager.cc:1428] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
I20260925 01:34:27.355999 16193 raft_consensus.cc:348] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.356554 16193 raft_consensus.cc:374] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:27.356693 16193 raft_consensus.cc:729] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0935a5422dc74024bbe9e6f7e5330b4c, State: Initialized, Role: FOLLOWER
I20260925 01:34:27.357440 16193 consensus_queue.cc:260] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.358817 16173 heartbeater.cc:503] Master 127.27.160.62:33983 was elected leader, sending a full tablet report...
I20260925 01:34:27.359370 16193 ts_tablet_manager.cc:1428] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260925 01:34:27.416870 16199 raft_consensus.cc:482] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:34:27.417330 16199 raft_consensus.cc:504] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.419325 16199 leader_election.cc:288] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 540b9975c27d42f7b0f8422a6c3f58ed (127.27.160.1:46557), cc73f57058024f0689710bb22763dbae (127.27.160.2:39803)
I20260925 01:34:27.426856 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.419995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:56703 (local address 127.27.160.1:46557)
0925 01:34:27.420223 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.420227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.420279 (+ 52us) server_negotiation.cc:407] Connection header received
0925 01:34:27.420487 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.420497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.420608 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.420752 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:27.421328 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.422548 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.424476 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.425705 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.425848 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.425904 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.426358 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.426490 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 01:34:27.426856 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.419702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.27.160.3:56703)
0925 01:34:27.420054 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.420095 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:27.420262 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:27.420789 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:27.420792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:27.420817 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:27.421101 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.421143 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.422704 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.422710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.424283 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.424296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.425891 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.425898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.426146 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.426191 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 01:34:27.426309 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 01:34:27.426472 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260925 01:34:27.427474 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.419928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.2:39803 (local address 127.27.160.3:47135)
0925 01:34:27.420798 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.420819 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:34:27.420965 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:34:27.421521 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:34:27.421528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:34:27.421558 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:34:27.421959 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.421972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.423210 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.423214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.424997 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:34:27.425012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.426587 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.426593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:34:27.426791 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.426826 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:34:27.426941 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 01:34:27.427151 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":523,"threads_started":1}
I20260925 01:34:27.427460 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.419878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.160.3:47135 (local address 127.27.160.2:39803)
0925 01:34:27.420113 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.420120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.420930 (+ 810us) server_negotiation.cc:407] Connection header received
0925 01:34:27.421161 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.421169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.421283 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.421371 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:34:27.422159 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.423017 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.425272 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.426400 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.426580 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.426668 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.426996 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.427146 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 01:34:27.428521 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6986495ad490899294215a83a99ff" candidate_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" is_pre_election: true
I20260925 01:34:27.428660 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6986495ad490899294215a83a99ff" candidate_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc73f57058024f0689710bb22763dbae" is_pre_election: true
I20260925 01:34:27.429275 15998 raft_consensus.cc:2454] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0935a5422dc74024bbe9e6f7e5330b4c in term 0.
I20260925 01:34:27.429275 15867 raft_consensus.cc:2454] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0935a5422dc74024bbe9e6f7e5330b4c in term 0.
I20260925 01:34:27.430104 16064 leader_election.cc:302] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0935a5422dc74024bbe9e6f7e5330b4c, cc73f57058024f0689710bb22763dbae; no voters:
I20260925 01:34:27.430658 16199 raft_consensus.cc:2781] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:34:27.430830 16199 raft_consensus.cc:482] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:34:27.430955 16199 raft_consensus.cc:3037] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:27.433327 16199 raft_consensus.cc:504] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.434000 16199 leader_election.cc:288] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [CANDIDATE]: Term 1 election: Requested vote from peers 540b9975c27d42f7b0f8422a6c3f58ed (127.27.160.1:46557), cc73f57058024f0689710bb22763dbae (127.27.160.2:39803)
I20260925 01:34:27.434620 15867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6986495ad490899294215a83a99ff" candidate_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540b9975c27d42f7b0f8422a6c3f58ed"
I20260925 01:34:27.434697 15998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9d6986495ad490899294215a83a99ff" candidate_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc73f57058024f0689710bb22763dbae"
I20260925 01:34:27.434883 15867 raft_consensus.cc:3037] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:27.434916 15998 raft_consensus.cc:3037] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:27.438611 15867 raft_consensus.cc:2454] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0935a5422dc74024bbe9e6f7e5330b4c in term 1.
I20260925 01:34:27.438620 15998 raft_consensus.cc:2454] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0935a5422dc74024bbe9e6f7e5330b4c in term 1.
I20260925 01:34:27.439167 16064 leader_election.cc:302] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0935a5422dc74024bbe9e6f7e5330b4c, cc73f57058024f0689710bb22763dbae; no voters:
I20260925 01:34:27.439515 16199 raft_consensus.cc:2781] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:27.440099 16199 raft_consensus.cc:686] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 1 LEADER]: Becoming Leader. State: Replica: 0935a5422dc74024bbe9e6f7e5330b4c, State: Running, Role: LEADER
I20260925 01:34:27.440642 16199 consensus_queue.cc:237] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } }
I20260925 01:34:27.446215 15725 catalog_manager.cc:5495] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0935a5422dc74024bbe9e6f7e5330b4c (127.27.160.3). New cstate: current_term: 1 leader_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0935a5422dc74024bbe9e6f7e5330b4c" member_type: VOTER last_known_addr { host: "127.27.160.3" port: 46201 } health_report { overall_health: HEALTHY } } }
W20260925 01:34:27.465963 16174 tablet.cc:2369] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:34:27.483527 16044 tablet.cc:2369] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:34:27.490890 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.484536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.27.160.3:46201)
0925 01:34:27.484753 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.484760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.484873 (+ 113us) server_negotiation.cc:407] Connection header received
0925 01:34:27.485176 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.485182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.485281 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.485457 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:34:27.485883 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.486699 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.488283 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.488752 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.488907 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.489529 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:34:27.490083 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:34:27.490273 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.490363 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.490467 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:34:27.500679 15998 raft_consensus.cc:1261] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 1 FOLLOWER]: Refusing update from remote peer 0935a5422dc74024bbe9e6f7e5330b4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:27.500705 15867 raft_consensus.cc:1261] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 1 FOLLOWER]: Refusing update from remote peer 0935a5422dc74024bbe9e6f7e5330b4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:34:27.501884 16201 consensus_queue.cc:1035] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "540b9975c27d42f7b0f8422a6c3f58ed" member_type: VOTER last_known_addr { host: "127.27.160.1" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:34:27.502257 16199 consensus_queue.cc:1035] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc73f57058024f0689710bb22763dbae" member_type: VOTER last_known_addr { host: "127.27.160.2" port: 39803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:34:27.511988 15913 tablet.cc:2369] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:34:27.512254 16204 mvcc.cc:205] Tried to move back new op lower bound from 7333069076471672832 to 7333069076241268736. Current Snapshot: MvccSnapshot[applied={T|T < 7333069076471672832}]
I20260925 01:34:27.541513 15726 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:27.541887 15726 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:34:27.547498 15726 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d8cce2438334cde8358610c7363d189: Sending DeleteTablet for 3 replicas of tablet b9d6986495ad490899294215a83a99ff
I20260925 01:34:27.548990 15978 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d6986495ad490899294215a83a99ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:27 UTC) from {username='slave'} at 127.0.0.1:55988
I20260925 01:34:27.549458 15847 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d6986495ad490899294215a83a99ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:27 UTC) from {username='slave'} at 127.0.0.1:48182
I20260925 01:34:27.549985 16108 tablet_service.cc:1514] Processing DeleteTablet for tablet b9d6986495ad490899294215a83a99ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:34:27 UTC) from {username='slave'} at 127.0.0.1:51226
I20260925 01:34:27.550956 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 15785
I20260925 01:34:27.562155 16214 tablet_replica.cc:330] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: stopping tablet replica
I20260925 01:34:27.563326 16215 tablet_replica.cc:330] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: stopping tablet replica
I20260925 01:34:27.564216 16214 raft_consensus.cc:2229] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:27.564217 16215 raft_consensus.cc:2229] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:34:27.564811 16214 raft_consensus.cc:2258] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:27.564815 16215 raft_consensus.cc:2258] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:27.566655 16216 tablet_replica.cc:330] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: stopping tablet replica
I20260925 01:34:27.567135 16216 raft_consensus.cc:2229] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:34:27.567423 16214 ts_tablet_manager.cc:1905] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:27.567514 16215 ts_tablet_manager.cc:1905] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:27.568372 16216 raft_consensus.cc:2258] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:34:27.570890 16216 ts_tablet_manager.cc:1905] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:34:27.575626 16215 ts_tablet_manager.cc:1918] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:27.575623 16214 ts_tablet_manager.cc:1918] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:27.575875 16214 log.cc:1196] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-1/wal/wals/b9d6986495ad490899294215a83a99ff
I20260925 01:34:27.576347 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.555271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.27.160.1:46557)
0925 01:34:27.555577 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.555589 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.567417 (+ 11828us) server_negotiation.cc:407] Connection header received
0925 01:34:27.567966 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.567972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.568071 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.568160 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:27.568895 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.570113 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.572004 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.572678 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.572796 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.574464 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:27.574476 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:27.574495 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:27.575276 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:27.575341 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:27.575370 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:27.575461 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.575738 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.575862 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:34:27.576694 16214 ts_tablet_manager.cc:1939] T b9d6986495ad490899294215a83a99ff P cc73f57058024f0689710bb22763dbae: Deleting consensus metadata
I20260925 01:34:27.575932 16215 log.cc:1196] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-0/wal/wals/b9d6986495ad490899294215a83a99ff
I20260925 01:34:27.577243 15907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:27.578984 15713 catalog_manager.cc:4841] TS cc73f57058024f0689710bb22763dbae (127.27.160.2:39803): tablet b9d6986495ad490899294215a83a99ff (table test-table [id=b0f5c72f374d4ccd99dacab414d34c35]) successfully deleted
I20260925 01:34:27.586995 16216 ts_tablet_manager.cc:1918] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:34:27.587239 16216 log.cc:1196] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Deleting WAL directory at /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790299737515179-28288-0/minicluster-data/ts-2/wal/wals/b9d6986495ad490899294215a83a99ff
I20260925 01:34:27.587873 16216 ts_tablet_manager.cc:1939] T b9d6986495ad490899294215a83a99ff P 0935a5422dc74024bbe9e6f7e5330b4c: Deleting consensus metadata
I20260925 01:34:27.590128 15711 catalog_manager.cc:4841] TS 0935a5422dc74024bbe9e6f7e5330b4c (127.27.160.3:46201): tablet b9d6986495ad490899294215a83a99ff (table test-table [id=b0f5c72f374d4ccd99dacab414d34c35]) successfully deleted
I20260925 01:34:27.683305 16215 ts_tablet_manager.cc:1939] T b9d6986495ad490899294215a83a99ff P 540b9975c27d42f7b0f8422a6c3f58ed: Deleting consensus metadata
I20260925 01:34:27.684072 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 15785
W20260925 01:34:27.702076 15711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.160.1:46557 (error 108)
I20260925 01:34:27.703542 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 15916
I20260925 01:34:27.704073 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.703564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48190)
0925 01:34:27.703740 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.703875 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 01:34:27.704419 15711 catalog_manager.cc:4584] TS 540b9975c27d42f7b0f8422a6c3f58ed (127.27.160.1:46557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9d6986495ad490899294215a83a99ff: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
I20260925 01:34:27.712015 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.704502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.27.160.2:39803)
0925 01:34:27.704708 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.704716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.704740 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:34:27.704826 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.704833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.704951 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.705072 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:27.705526 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.706852 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.708154 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.708895 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.709036 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.710240 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:27.710254 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:27.710277 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:27.711126 (+ 849us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:27.711191 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:27.711224 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:27.711349 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.711469 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.711600 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:34:27.712908 16038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:27.758343 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.757733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48206)
0925 01:34:27.757989 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.758125 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:34:27.759588 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.759161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48208)
0925 01:34:27.759337 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.759429 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 01:34:27.806052 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 15916
I20260925 01:34:27.820394 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.819921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48212)
0925 01:34:27.820129 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.820237 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 01:34:27.821633 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.821255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48224)
0925 01:34:27.821417 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.821502 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 01:34:27.829926 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16046
I20260925 01:34:27.838516 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.830857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.27.160.3:46201)
0925 01:34:27.831054 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:27.831060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:27.831083 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:34:27.831296 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:27.831302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:27.831410 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:27.831539 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:27.831949 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.833232 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.834598 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:27.835301 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:27.835446 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:27.836729 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:27.836740 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:27.836756 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:27.837636 (+ 880us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:27.837698 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:27.837726 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:27.837834 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:27.837992 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 01:34:27.838102 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:34:27.839440 16168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:27.902390 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.901746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48228)
0925 01:34:27.902002 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.902179 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 01:34:27.903858 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:27.903324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48240)
0925 01:34:27.903566 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:27.903701 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 01:34:27.961172 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 16046
I20260925 01:34:27.980620 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 15695
I20260925 01:34:27.981957 15756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:28.095391 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:28.094570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.1:46557 (local address 127.0.0.1:48256)
0925 01:34:28.094887 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 01:34:28.095016 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.1:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260925 01:34:28.095779 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 15695
2026-09-25T01:34:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2981 ms)
[----------] 6 tests from EncryptionPolicyTest (23401 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:28 dist-test-slave-h6hf krb5kdc[16225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:28 dist-test-slave-h6hf krb5kdc[16225](info): set up 2 sockets
Sep 25 01:34:28 dist-test-slave-h6hf krb5kdc[16225](info): commencing operation
krb5kdc: starting...
W20260925 01:34:29.274734 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:29 dist-test-slave-h6hf krb5kdc[16225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:29.557857 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:29.919938 16238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:29.920275 16238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:29.920380 16238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:29.927914 16238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:29.928064 16238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:29.928119 16238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:29.928190 16238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:29.928243 16238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:29.938658 16238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:29.940111 16238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:29.941547 16238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:29.954931 16246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:29.954890 16244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:29.955281 16243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:29.956120 16238 server_base.cc:1034] running on GCE node
I20260925 01:34:29.957170 16238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:29.959036 16238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:29.960281 16238 hybrid_clock.cc:648] HybridClock initialized: now 1790300069960228 us; error 57 us; skew 500 ppm
I20260925 01:34:29.960666 16238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:29 dist-test-slave-h6hf krb5kdc[16225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:29.968855 16238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:29.970808 16238 webserver.cc:460] Webserver started at http://127.0.0.1:45669/ using document root <none> and password file <none>
I20260925 01:34:29.971626 16238 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:29.971803 16238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:29.972159 16238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:29.975642 16238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "c8d1d6873eed4d4896ba376efd645cc3"
format_stamp: "Formatted at 2026-09-25 01:34:29 on dist-test-slave-h6hf"
server_key: "e3cfa542828763bdef8576e0605da69f"
server_key_iv: "fd3b6bbefda912ea27aaa6a12b420efb"
server_key_version: "encryptionkey@0"
I20260925 01:34:29.976529 16238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "c8d1d6873eed4d4896ba376efd645cc3"
format_stamp: "Formatted at 2026-09-25 01:34:29 on dist-test-slave-h6hf"
server_key: "e3cfa542828763bdef8576e0605da69f"
server_key_iv: "fd3b6bbefda912ea27aaa6a12b420efb"
server_key_version: "encryptionkey@0"
I20260925 01:34:29.982692 16238 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 01:34:29.987315 16253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:29.988850 16238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:34:29.989094 16238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "c8d1d6873eed4d4896ba376efd645cc3"
format_stamp: "Formatted at 2026-09-25 01:34:29 on dist-test-slave-h6hf"
server_key: "e3cfa542828763bdef8576e0605da69f"
server_key_iv: "fd3b6bbefda912ea27aaa6a12b420efb"
server_key_version: "encryptionkey@0"
I20260925 01:34:29.989423 16238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:30.024298 16238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:30.029654 16238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:30.030018 16238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:30.051131 16238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37377
I20260925 01:34:30.051196 16304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37377 every 8 connection(s)
I20260925 01:34:30.052686 16238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:30.056898 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 16238
I20260925 01:34:30.057170 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:30.057332 16305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:30.058087 28288 external_mini_cluster.cc:1428] Setting key c9e58f68a8ad4997c5af5cca4a778cb5
I20260925 01:34:30.074563 16305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3: Bootstrap starting.
Sep 25 01:34:30 dist-test-slave-h6hf krb5kdc[16225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:34:30.078871 16305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:30.080464 16305 log.cc:824] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:30.084374 16305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3: No bootstrap required, opened a new log
I20260925 01:34:30.086946 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:30.060313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43225 (local address 127.0.0.1:37377)
0925 01:34:30.061372 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:30.061389 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:30.061439 (+ 50us) server_negotiation.cc:407] Connection header received
0925 01:34:30.063226 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:30.063249 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:30.063734 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:30.064122 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:30.066309 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:30.067837 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:30.070005 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:30.070867 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:30.071028 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:30.076949 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:30.076958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:30.076974 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:30.077023 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:30.081463 (+ 4440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:30.082604 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:30.082613 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:30.082622 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:30.082851 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:30.083352 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:30.083357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:30.083360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:30.084109 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:30.084489 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:30.085029 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 01:34:30.085466 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":497,"threads_started":1}
I20260925 01:34:30.091506 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:30.094615 16305 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d1d6873eed4d4896ba376efd645cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260925 01:34:30.095185 16305 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:30.095330 16305 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d1d6873eed4d4896ba376efd645cc3, State: Initialized, Role: FOLLOWER
I20260925 01:34:30.096081 16305 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d1d6873eed4d4896ba376efd645cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260925 01:34:30.096381 16305 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:30.096552 16305 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:30.096800 16305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:30.100059 16305 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d1d6873eed4d4896ba376efd645cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260925 01:34:30.100744 16305 leader_election.cc:302] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8d1d6873eed4d4896ba376efd645cc3; no voters:
I20260925 01:34:30.101605 16305 leader_election.cc:288] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:30.101723 16315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:30.102924 16315 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [term 1 LEADER]: Becoming Leader. State: Replica: c8d1d6873eed4d4896ba376efd645cc3, State: Running, Role: LEADER
I20260925 01:34:30.103559 16315 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d1d6873eed4d4896ba376efd645cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } }
I20260925 01:34:30.104220 16305 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:30.106779 16316 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8d1d6873eed4d4896ba376efd645cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d1d6873eed4d4896ba376efd645cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } } }
I20260925 01:34:30.107312 16316 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:30.108477 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8d1d6873eed4d4896ba376efd645cc3. Latest consensus state: current_term: 1 leader_uuid: "c8d1d6873eed4d4896ba376efd645cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8d1d6873eed4d4896ba376efd645cc3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37377 } } }
I20260925 01:34:30.109099 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:30.112710 16320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:30.115566 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:30.096579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56151 (local address 127.0.0.1:37377)
0925 01:34:30.096867 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:30.096875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:30.097355 (+ 480us) server_negotiation.cc:407] Connection header received
0925 01:34:30.097496 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:30.097504 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:30.097603 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:30.097740 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:30.099451 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:30.100675 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:30.102591 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:30.103336 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:30.103489 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:30.106699 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:30.106709 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:30.106718 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:30.106765 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:30.111387 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:30.112528 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:30.112534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:30.112537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:30.112730 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:30.113451 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:30.113460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:30.113462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:30.113927 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:30.114225 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:30.114927 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 01:34:30.115208 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260925 01:34:30.119056 16320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:30.129194 16320 catalog_manager.cc:1347] Generated new cluster ID: 4f2fdca412a540288f40025477591f44
I20260925 01:34:30.129340 16320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:30.150892 16320 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:30.152434 16320 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:30.167009 16320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8d1d6873eed4d4896ba376efd645cc3: Generated new TSK 0
I20260925 01:34:30.167935 16320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:30.181259 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16238
I20260925 01:34:30.183192 16300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:30.280609 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 16238
W20260925 01:34:30.295766 16223 connection.cc:535] client connection to 127.0.0.1:37377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37377): Connection reset by peer (error 104)
2026-09-25T01:34:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2172 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:30 dist-test-slave-h6hf krb5kdc[16337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:30 dist-test-slave-h6hf krb5kdc[16337](info): set up 2 sockets
Sep 25 01:34:30 dist-test-slave-h6hf krb5kdc[16337](info): commencing operation
krb5kdc: starting...
W20260925 01:34:31.446645 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:31 dist-test-slave-h6hf krb5kdc[16337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:31.728456 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:32.072976 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:32.073350 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:32.073462 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:32.080845 16350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:32.080969 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:32.081023 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:32.081104 16350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:32.081174 16350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:32.091342 16350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:32.092643 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:32.093959 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:32.106189 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:32.106171 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:32.106221 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:32.107111 16350 server_base.cc:1034] running on GCE node
I20260925 01:34:32.107985 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:32.109649 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:32.110927 16350 hybrid_clock.cc:648] HybridClock initialized: now 1790300072110869 us; error 39 us; skew 500 ppm
I20260925 01:34:32.111389 16350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:32 dist-test-slave-h6hf krb5kdc[16337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:32.114966 16350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:32.116631 16350 webserver.cc:460] Webserver started at http://127.0.0.1:40803/ using document root <none> and password file <none>
I20260925 01:34:32.117406 16350 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:32.117548 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:32.117857 16350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:32.120963 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7"
format_stamp: "Formatted at 2026-09-25 01:34:32 on dist-test-slave-h6hf"
server_key: "6b3ef14187a4b75672ed52505240fdad"
server_key_iv: "76c1b3210d6173f5c2b9740be4d5eef5"
server_key_version: "encryptionkey@0"
I20260925 01:34:32.121814 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7"
format_stamp: "Formatted at 2026-09-25 01:34:32 on dist-test-slave-h6hf"
server_key: "6b3ef14187a4b75672ed52505240fdad"
server_key_iv: "76c1b3210d6173f5c2b9740be4d5eef5"
server_key_version: "encryptionkey@0"
I20260925 01:34:32.127384 16350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 01:34:32.131323 16365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:32.132805 16350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:34:32.133011 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7"
format_stamp: "Formatted at 2026-09-25 01:34:32 on dist-test-slave-h6hf"
server_key: "6b3ef14187a4b75672ed52505240fdad"
server_key_iv: "76c1b3210d6173f5c2b9740be4d5eef5"
server_key_version: "encryptionkey@0"
I20260925 01:34:32.133335 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:32.147372 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:32.152110 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:32.152413 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:32.171554 16350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42959
I20260925 01:34:32.171664 16416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42959 every 8 connection(s)
I20260925 01:34:32.172997 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:32.175324 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 16350
I20260925 01:34:32.175559 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:32.176152 28288 external_mini_cluster.cc:1428] Setting key 4114db6bad8e9d7c58c7787a786ad787
I20260925 01:34:32.178303 16417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:32.190476 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:32.178046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46381 (local address 127.0.0.1:42959)
0925 01:34:32.178880 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:32.178893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:32.178936 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:34:32.180611 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:32.180632 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:32.181062 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:32.181510 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:32.182242 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:32.183634 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:32.185739 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:32.186435 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:32.186577 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:32.188012 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:32.188019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:32.188034 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:32.188627 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:32.188706 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:32.188736 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:32.188852 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:32.189264 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 01:34:32.189474 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":386,"threads_started":1}
I20260925 01:34:32.192477 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7: Bootstrap starting.
I20260925 01:34:32.194643 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:32.196195 16417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:32.197399 16417 log.cc:824] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:32.201012 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7: No bootstrap required, opened a new log
I20260925 01:34:32.206228 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:32.199063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52673 (local address 127.0.0.1:42959)
0925 01:34:32.199250 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:32.199257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:32.199602 (+ 345us) server_negotiation.cc:407] Connection header received
0925 01:34:32.199765 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:32.199771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:32.199868 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:32.199989 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:32.200451 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:32.201714 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:32.203109 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:32.204044 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:32.204204 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:32.205375 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:32.205383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:32.205388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:32.205411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:32.205465 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:32.205494 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:32.205642 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:32.205786 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 01:34:32.205841 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:34:32.210937 16417 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } }
I20260925 01:34:32.211337 16417 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:32.211416 16417 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d6b5ff1e964b04b72b3b53c9c64bf7, State: Initialized, Role: FOLLOWER
I20260925 01:34:32.212033 16417 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } }
I20260925 01:34:32.212291 16417 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:32.212468 16417 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:32.212694 16417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:32.215307 16417 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } }
I20260925 01:34:32.215902 16417 leader_election.cc:302] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5d6b5ff1e964b04b72b3b53c9c64bf7; no voters:
I20260925 01:34:32.216537 16417 leader_election.cc:288] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:32.216692 16427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:32.217538 16427 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [term 1 LEADER]: Becoming Leader. State: Replica: d5d6b5ff1e964b04b72b3b53c9c64bf7, State: Running, Role: LEADER
I20260925 01:34:32.218139 16427 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } }
I20260925 01:34:32.218967 16417 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:32.221043 16428 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } } }
I20260925 01:34:32.221447 16428 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:32.222322 16435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:32.224244 16429 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5d6b5ff1e964b04b72b3b53c9c64bf7. Latest consensus state: current_term: 1 leader_uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d6b5ff1e964b04b72b3b53c9c64bf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42959 } } }
I20260925 01:34:32.224680 16429 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:32.232959 16435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:32.244016 16435 catalog_manager.cc:1347] Generated new cluster ID: 0ce7ab6bfbf74f00a61a4a9d64eff5a3
I20260925 01:34:32.244184 16435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:32.280597 16435 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:32.282186 16435 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:32.291353 16435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5d6b5ff1e964b04b72b3b53c9c64bf7: Generated new TSK 0
I20260925 01:34:32.292438 16435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:32.310695 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16350
W20260925 01:34:32.311863 16412 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46381
W20260925 01:34:32.312942 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16350: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:32.313179 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 16350
W20260925 01:34:32.332165 16335 connection.cc:535] client connection to 127.0.0.1:42959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42959): Connection reset by peer (error 104)
2026-09-25T01:34:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2035 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:32 dist-test-slave-h6hf krb5kdc[16448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:32 dist-test-slave-h6hf krb5kdc[16448](info): set up 2 sockets
Sep 25 01:34:32 dist-test-slave-h6hf krb5kdc[16448](info): commencing operation
krb5kdc: starting...
W20260925 01:34:33.467526 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:33 dist-test-slave-h6hf krb5kdc[16448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:33.745234 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:34.109910 16461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:34.110219 16461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:34.110313 16461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:34.118040 16461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:34.118176 16461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:34.118235 16461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:34.118299 16461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:34.118342 16461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:34.128648 16461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:34.130050 16461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:34.131363 16461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:34.144364 16466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:34.144399 16467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:34.144433 16469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:34.145258 16461 server_base.cc:1034] running on GCE node
I20260925 01:34:34.146144 16461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:34.147673 16461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:34.148919 16461 hybrid_clock.cc:648] HybridClock initialized: now 1790300074148852 us; error 60 us; skew 500 ppm
I20260925 01:34:34.149369 16461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:34 dist-test-slave-h6hf krb5kdc[16448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:34.152789 16461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:34.154331 16461 webserver.cc:460] Webserver started at http://127.0.0.1:40725/ using document root <none> and password file <none>
I20260925 01:34:34.155040 16461 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:34.155184 16461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:34.155490 16461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:34.158653 16461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "65b867d2471c45198e3bf0fd276db425"
format_stamp: "Formatted at 2026-09-25 01:34:34 on dist-test-slave-h6hf"
server_key: "707fe87f67f00d291354d06acb0bdd37"
server_key_iv: "3c95c4db364e9612a12c38b0b383f304"
server_key_version: "encryptionkey@0"
I20260925 01:34:34.159494 16461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "65b867d2471c45198e3bf0fd276db425"
format_stamp: "Formatted at 2026-09-25 01:34:34 on dist-test-slave-h6hf"
server_key: "707fe87f67f00d291354d06acb0bdd37"
server_key_iv: "3c95c4db364e9612a12c38b0b383f304"
server_key_version: "encryptionkey@0"
I20260925 01:34:34.165194 16461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:34:34.169052 16476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:34.170652 16461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 01:34:34.170856 16461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "65b867d2471c45198e3bf0fd276db425"
format_stamp: "Formatted at 2026-09-25 01:34:34 on dist-test-slave-h6hf"
server_key: "707fe87f67f00d291354d06acb0bdd37"
server_key_iv: "3c95c4db364e9612a12c38b0b383f304"
server_key_version: "encryptionkey@0"
I20260925 01:34:34.171067 16461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:34.177027 16461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:34.181718 16461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:34.182036 16461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:34.200598 16461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34897
I20260925 01:34:34.200635 16527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34897 every 8 connection(s)
I20260925 01:34:34.202349 16461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:34.206238 16528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:34.212592 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 16461
I20260925 01:34:34.212776 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:34.213301 28288 external_mini_cluster.cc:1428] Setting key 5a55c2554dda2703397efa40e121f71d
I20260925 01:34:34.217455 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425: Bootstrap starting.
I20260925 01:34:34.220846 16528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:34.221276 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:34.215231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33125 (local address 127.0.0.1:34897)
0925 01:34:34.216016 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:34.216027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:34.216073 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:34:34.217846 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:34.217871 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:34.218214 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:34.218588 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:34.219633 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:34.219639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:34.219651 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:34.219664 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:34.219728 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:34.219752 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:34.219971 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:34.220107 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 01:34:34.220280 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":339,"threads_started":1}
I20260925 01:34:34.222245 16528 log.cc:824] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:34.225203 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:34.225577 16528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425: No bootstrap required, opened a new log
I20260925 01:34:34.233074 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:34.230560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36591 (local address 127.0.0.1:34897)
0925 01:34:34.230757 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:34.230764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:34.231178 (+ 414us) server_negotiation.cc:407] Connection header received
0925 01:34:34.231338 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:34.231344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:34.231411 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:34.231516 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:34.232427 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:34.232432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:34.232437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:34.232455 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:34.232507 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:34.232526 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:34.232664 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:34.232734 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 01:34:34.232787 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:34:34.235641 16528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b867d2471c45198e3bf0fd276db425" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260925 01:34:34.236001 16528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:34.236075 16528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b867d2471c45198e3bf0fd276db425, State: Initialized, Role: FOLLOWER
I20260925 01:34:34.236586 16528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b867d2471c45198e3bf0fd276db425" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260925 01:34:34.236797 16528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:34.236899 16528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:34.237028 16528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:34.239266 16528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b867d2471c45198e3bf0fd276db425" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260925 01:34:34.239735 16528 leader_election.cc:302] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65b867d2471c45198e3bf0fd276db425; no voters:
I20260925 01:34:34.240411 16528 leader_election.cc:288] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:34.240541 16538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:34.247748 16538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [term 1 LEADER]: Becoming Leader. State: Replica: 65b867d2471c45198e3bf0fd276db425, State: Running, Role: LEADER
I20260925 01:34:34.248311 16538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b867d2471c45198e3bf0fd276db425" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } }
I20260925 01:34:34.251000 16528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:34.256961 16539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65b867d2471c45198e3bf0fd276db425. Latest consensus state: current_term: 1 leader_uuid: "65b867d2471c45198e3bf0fd276db425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b867d2471c45198e3bf0fd276db425" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } } }
I20260925 01:34:34.257539 16539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:34.258551 16543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:34.259166 16540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65b867d2471c45198e3bf0fd276db425" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b867d2471c45198e3bf0fd276db425" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34897 } } }
I20260925 01:34:34.259518 16540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:34.264175 16543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:34.274150 16543 catalog_manager.cc:1347] Generated new cluster ID: f01fc74effba4a38b33704f918cefad4
I20260925 01:34:34.274290 16543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:34.289664 16543 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:34.290136 16543 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:34.300629 16543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65b867d2471c45198e3bf0fd276db425: Generated new TSK 0
I20260925 01:34:34.301692 16543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:34.312633 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16461
W20260925 01:34:34.313711 16523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33125
W20260925 01:34:34.314400 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16461: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:34.314553 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 16461
2026-09-25T01:34:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1999 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:34 dist-test-slave-h6hf krb5kdc[16559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:34 dist-test-slave-h6hf krb5kdc[16559](info): set up 2 sockets
Sep 25 01:34:34 dist-test-slave-h6hf krb5kdc[16559](info): commencing operation
krb5kdc: starting...
W20260925 01:34:35.539372 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:35 dist-test-slave-h6hf krb5kdc[16559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:35.816488 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:36.156387 16572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:36.156733 16572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:36.156890 16572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:36.164403 16572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:36.164520 16572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:36.164636 16572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:36.164736 16572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:36.164811 16572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:36.175225 16572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:36.176743 16572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:36.178393 16572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:36.190893 16580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:36.190918 16578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:36.190965 16577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:36.191818 16572 server_base.cc:1034] running on GCE node
I20260925 01:34:36.192624 16572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:36.194149 16572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:36.195371 16572 hybrid_clock.cc:648] HybridClock initialized: now 1790300076195301 us; error 55 us; skew 500 ppm
I20260925 01:34:36.195744 16572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:36 dist-test-slave-h6hf krb5kdc[16559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:36.207903 16572 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:36.209721 16572 webserver.cc:460] Webserver started at http://127.0.0.1:46159/ using document root <none> and password file <none>
I20260925 01:34:36.210505 16572 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:36.210676 16572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:36.210991 16572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:36.214236 16572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "42394d8c000841cf9cdc8670dea36ac2"
format_stamp: "Formatted at 2026-09-25 01:34:36 on dist-test-slave-h6hf"
server_key: "15af7c136e7ec67875c71ecbd5bb3993"
server_key_iv: "204e814592d1938c49e494e964b11355"
server_key_version: "encryptionkey@0"
I20260925 01:34:36.215051 16572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "42394d8c000841cf9cdc8670dea36ac2"
format_stamp: "Formatted at 2026-09-25 01:34:36 on dist-test-slave-h6hf"
server_key: "15af7c136e7ec67875c71ecbd5bb3993"
server_key_iv: "204e814592d1938c49e494e964b11355"
server_key_version: "encryptionkey@0"
I20260925 01:34:36.220518 16572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:34:36.224303 16587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:36.225891 16572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:36.226080 16572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "42394d8c000841cf9cdc8670dea36ac2"
format_stamp: "Formatted at 2026-09-25 01:34:36 on dist-test-slave-h6hf"
server_key: "15af7c136e7ec67875c71ecbd5bb3993"
server_key_iv: "204e814592d1938c49e494e964b11355"
server_key_version: "encryptionkey@0"
I20260925 01:34:36.226289 16572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:36.250689 16572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:36.255429 16572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:36.255738 16572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:36.276137 16572 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45659
I20260925 01:34:36.276168 16638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45659 every 8 connection(s)
I20260925 01:34:36.277662 16572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:36.281490 16639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:36.285375 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 16572
I20260925 01:34:36.285562 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:36.286011 28288 external_mini_cluster.cc:1428] Setting key 3f8556394454ec525fed34e1ff9113b9
I20260925 01:34:36.295253 16639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2: Bootstrap starting.
I20260925 01:34:36.298942 16639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:36.300038 16639 log.cc:824] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:34:36 dist-test-slave-h6hf krb5kdc[16559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:34:36.303359 16639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2: No bootstrap required, opened a new log
I20260925 01:34:36.310369 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:36.288025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42047 (local address 127.0.0.1:45659)
0925 01:34:36.288910 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:36.288926 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:36.288977 (+ 51us) server_negotiation.cc:407] Connection header received
0925 01:34:36.290859 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:36.290891 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:36.291467 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:36.291928 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:36.293246 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:36.295212 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:36.296614 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:36.297757 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:36.297954 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:36.302576 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:36.302585 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:36.302601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:36.302638 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:36.306011 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:36.306903 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:36.306910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:36.306916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:36.307088 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:36.307516 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:36.307519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:36.307521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:36.308070 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:36.308306 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:36.308796 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 01:34:36.309213 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":394,"threads_started":1}
I20260925 01:34:36.312810 16639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42394d8c000841cf9cdc8670dea36ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45659 } }
I20260925 01:34:36.313336 16639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:36.313443 16639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42394d8c000841cf9cdc8670dea36ac2, State: Initialized, Role: FOLLOWER
I20260925 01:34:36.314057 16639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42394d8c000841cf9cdc8670dea36ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45659 } }
I20260925 01:34:36.314360 16639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:36.314409 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:36.314518 16639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:36.314685 16639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:36.316867 16639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42394d8c000841cf9cdc8670dea36ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45659 } }
I20260925 01:34:36.317458 16639 leader_election.cc:302] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42394d8c000841cf9cdc8670dea36ac2; no voters:
I20260925 01:34:36.318115 16639 leader_election.cc:288] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:36.318231 16648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:36.320577 16648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 42394d8c000841cf9cdc8670dea36ac2, State: Running, Role: LEADER
I20260925 01:34:36.320734 16639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:36.321274 16648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42394d8c000841cf9cdc8670dea36ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45659 } }
I20260925 01:34:36.324921 16649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42394d8c000841cf9cdc8670dea36ac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42394d8c000841cf9cdc8670dea36ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45659 } } }
I20260925 01:34:36.325417 16649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:36.326273 16650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42394d8c000841cf9cdc8670dea36ac2. Latest consensus state: current_term: 1 leader_uuid: "42394d8c000841cf9cdc8670dea36ac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42394d8c000841cf9cdc8670dea36ac2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45659 } } }
I20260925 01:34:36.326666 16650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:36.330750 16658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:36.338177 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:36.319509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47733 (local address 127.0.0.1:45659)
0925 01:34:36.319897 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:36.319902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:36.319916 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:34:36.319979 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:36.319985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:36.320053 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:36.320143 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:36.321891 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:36.323187 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:36.325002 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:36.325735 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:36.325865 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:36.329835 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:36.329843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:36.329851 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:36.329896 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:36.333780 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:36.334762 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:36.334767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:36.334770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:36.334927 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:36.335436 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:36.335443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:36.335448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:36.335884 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:36.336134 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:36.337556 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0925 01:34:36.337788 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260925 01:34:36.338220 16658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:36.348646 16658 catalog_manager.cc:1347] Generated new cluster ID: b7522d85c16c48a799e0369aa5165b1c
I20260925 01:34:36.348781 16658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:36.370932 16658 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:36.372869 16658 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:36.380440 16658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42394d8c000841cf9cdc8670dea36ac2: Generated new TSK 0
I20260925 01:34:36.381513 16658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:36.388099 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16572
I20260925 01:34:36.389353 16634 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:36.480612 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 16572
2026-09-25T01:34:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2166 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:36 dist-test-slave-h6hf krb5kdc[16671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:36 dist-test-slave-h6hf krb5kdc[16671](info): set up 2 sockets
Sep 25 01:34:36 dist-test-slave-h6hf krb5kdc[16671](info): commencing operation
krb5kdc: starting...
W20260925 01:34:37.630673 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:37 dist-test-slave-h6hf krb5kdc[16671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:37.908478 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:38.256691 16684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:38.256995 16684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:38.257201 16684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:38.264602 16684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:38.264722 16684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:38.264840 16684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:38.264942 16684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:38.265020 16684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:38.276917 16684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:38.278347 16684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:38.279820 16684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:38.293428 16690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:38.293505 16692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:38.293464 16689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:38.294255 16684 server_base.cc:1034] running on GCE node
I20260925 01:34:38.295085 16684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:38.296571 16684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:38.297814 16684 hybrid_clock.cc:648] HybridClock initialized: now 1790300078297776 us; error 57 us; skew 500 ppm
I20260925 01:34:38.298177 16684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:38 dist-test-slave-h6hf krb5kdc[16671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:38.301455 16684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:38.302963 16684 webserver.cc:460] Webserver started at http://127.0.0.1:43719/ using document root <none> and password file <none>
I20260925 01:34:38.303651 16684 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:38.303769 16684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:38.304113 16684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:38.307394 16684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "af9372e136184b12ae3bbf456be512af"
format_stamp: "Formatted at 2026-09-25 01:34:38 on dist-test-slave-h6hf"
server_key: "19a51df6bccf9fa0b9cb98b3ff0225ad"
server_key_iv: "927f379a23981e560beb3c8a54426f15"
server_key_version: "encryptionkey@0"
I20260925 01:34:38.308187 16684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "af9372e136184b12ae3bbf456be512af"
format_stamp: "Formatted at 2026-09-25 01:34:38 on dist-test-slave-h6hf"
server_key: "19a51df6bccf9fa0b9cb98b3ff0225ad"
server_key_iv: "927f379a23981e560beb3c8a54426f15"
server_key_version: "encryptionkey@0"
I20260925 01:34:38.313818 16684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:34:38.317962 16699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:38.319402 16684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:34:38.319550 16684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "af9372e136184b12ae3bbf456be512af"
format_stamp: "Formatted at 2026-09-25 01:34:38 on dist-test-slave-h6hf"
server_key: "19a51df6bccf9fa0b9cb98b3ff0225ad"
server_key_iv: "927f379a23981e560beb3c8a54426f15"
server_key_version: "encryptionkey@0"
I20260925 01:34:38.319736 16684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:38.340600 16684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:38.345024 16684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:38.345292 16684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:38.364004 16684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37403
I20260925 01:34:38.364048 16750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37403 every 8 connection(s)
I20260925 01:34:38.365579 16684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:38.369643 16751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:38.375653 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 16684
I20260925 01:34:38.375871 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:38.376365 28288 external_mini_cluster.cc:1428] Setting key 338f37dc96e5b58a93e1b299d5280f87
I20260925 01:34:38.381314 16751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af: Bootstrap starting.
I20260925 01:34:38.385113 16751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:38.386368 16751 log.cc:824] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:38.389410 16751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af: No bootstrap required, opened a new log
I20260925 01:34:38.390203 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:38.378286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58207 (local address 127.0.0.1:37403)
0925 01:34:38.379182 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:38.379198 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:38.379242 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:34:38.380904 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:38.380929 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:38.381422 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:38.381828 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:38.382483 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:38.383815 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:38.385769 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:38.386442 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:38.386645 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:38.387814 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:38.387822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:38.387837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:38.388393 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:38.388476 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:38.388504 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:38.388629 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:38.388853 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 01:34:38.389162 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":443,"threads_started":1}
I20260925 01:34:38.394153 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:38.397989 16751 raft_consensus.cc:348] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9372e136184b12ae3bbf456be512af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } }
I20260925 01:34:38.398495 16751 raft_consensus.cc:374] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:38.398641 16751 raft_consensus.cc:729] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9372e136184b12ae3bbf456be512af, State: Initialized, Role: FOLLOWER
I20260925 01:34:38.399339 16751 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9372e136184b12ae3bbf456be512af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } }
I20260925 01:34:38.399641 16751 raft_consensus.cc:388] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:38.399798 16751 raft_consensus.cc:482] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:38.399988 16751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:38.403192 16751 raft_consensus.cc:504] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9372e136184b12ae3bbf456be512af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } }
I20260925 01:34:38.403880 16751 leader_election.cc:302] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af9372e136184b12ae3bbf456be512af; no voters:
I20260925 01:34:38.404595 16751 leader_election.cc:288] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:38.404714 16761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:38.405859 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:38.398465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41957 (local address 127.0.0.1:37403)
0925 01:34:38.398662 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:38.398670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:38.399231 (+ 561us) server_negotiation.cc:407] Connection header received
0925 01:34:38.399418 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:38.399422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:38.399481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:38.399583 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:38.400258 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:38.401011 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:38.402909 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:38.403426 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:38.403531 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:38.404884 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:38.404893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:38.404901 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:38.404927 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:38.404980 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:38.405005 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:38.405105 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:38.405415 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:34:38.405520 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:34:38.406075 16761 raft_consensus.cc:686] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [term 1 LEADER]: Becoming Leader. State: Replica: af9372e136184b12ae3bbf456be512af, State: Running, Role: LEADER
I20260925 01:34:38.406840 16761 consensus_queue.cc:237] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9372e136184b12ae3bbf456be512af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } }
I20260925 01:34:38.407177 16751 sys_catalog.cc:564] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:38.411255 16763 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [sys.catalog]: SysCatalogTable state changed. Reason: New leader af9372e136184b12ae3bbf456be512af. Latest consensus state: current_term: 1 leader_uuid: "af9372e136184b12ae3bbf456be512af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9372e136184b12ae3bbf456be512af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } } }
I20260925 01:34:38.411249 16762 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af9372e136184b12ae3bbf456be512af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9372e136184b12ae3bbf456be512af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37403 } } }
I20260925 01:34:38.411755 16763 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:38.411759 16762 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:38.412945 16770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:38.421000 16770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:38.431162 16770 catalog_manager.cc:1347] Generated new cluster ID: 45570f593a8a44eabb8408c2c2e91bb4
I20260925 01:34:38.431300 16770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:38.452396 16770 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:38.453903 16770 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:38.467535 16770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af9372e136184b12ae3bbf456be512af: Generated new TSK 0
I20260925 01:34:38.468489 16770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:38.484436 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16684
W20260925 01:34:38.485872 16746 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58207
W20260925 01:34:38.486949 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:38.487147 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 16684
2026-09-25T01:34:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2006 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:38 dist-test-slave-h6hf krb5kdc[16782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:38 dist-test-slave-h6hf krb5kdc[16782](info): set up 2 sockets
Sep 25 01:34:38 dist-test-slave-h6hf krb5kdc[16782](info): commencing operation
krb5kdc: starting...
W20260925 01:34:39.634559 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:39 dist-test-slave-h6hf krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:39.913439 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:40.256037 16795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:40.256433 16795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:40.256613 16795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:40.264518 16795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:40.264719 16795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:40.264843 16795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:40.264950 16795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:40.265023 16795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:40.275305 16795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:40.276722 16795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:40.278069 16795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:40.291224 16800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:40.291178 16801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:40.291134 16803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:40.292421 16795 server_base.cc:1034] running on GCE node
I20260925 01:34:40.293480 16795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:40.295009 16795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:40.296277 16795 hybrid_clock.cc:648] HybridClock initialized: now 1790300080296222 us; error 49 us; skew 500 ppm
I20260925 01:34:40.296665 16795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:40 dist-test-slave-h6hf krb5kdc[16782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:40.300197 16795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:40.301893 16795 webserver.cc:460] Webserver started at http://127.0.0.1:39773/ using document root <none> and password file <none>
I20260925 01:34:40.302613 16795 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:40.302769 16795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:40.303088 16795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:40.306269 16795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d"
format_stamp: "Formatted at 2026-09-25 01:34:40 on dist-test-slave-h6hf"
server_key: "bfb940a0fcbca5b53172fb8210c1c506"
server_key_iv: "d4c21cbae4776193b9022644c7395f79"
server_key_version: "encryptionkey@0"
I20260925 01:34:40.307071 16795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d"
format_stamp: "Formatted at 2026-09-25 01:34:40 on dist-test-slave-h6hf"
server_key: "bfb940a0fcbca5b53172fb8210c1c506"
server_key_iv: "d4c21cbae4776193b9022644c7395f79"
server_key_version: "encryptionkey@0"
I20260925 01:34:40.312682 16795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260925 01:34:40.316445 16811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:40.318007 16795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:34:40.318213 16795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d"
format_stamp: "Formatted at 2026-09-25 01:34:40 on dist-test-slave-h6hf"
server_key: "bfb940a0fcbca5b53172fb8210c1c506"
server_key_iv: "d4c21cbae4776193b9022644c7395f79"
server_key_version: "encryptionkey@0"
I20260925 01:34:40.318444 16795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:40.324618 16795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:40.328910 16795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:40.329273 16795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:40.348297 16795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35463
I20260925 01:34:40.348353 16862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35463 every 8 connection(s)
I20260925 01:34:40.349938 16795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:40.353716 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:40.359772 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 16795
I20260925 01:34:40.359983 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:40.360433 28288 external_mini_cluster.cc:1428] Setting key 95936a8ad6968f9f1b58d1a83aebef2c
I20260925 01:34:40.364727 16863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d: Bootstrap starting.
I20260925 01:34:40.368523 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:40.362294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56185 (local address 127.0.0.1:35463)
0925 01:34:40.363215 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:40.363227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:40.363268 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:34:40.364981 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:40.365005 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:40.365420 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:40.365773 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:40.366916 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:40.366922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:40.366936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:40.366951 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:40.367018 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:40.367038 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:40.367205 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:40.367312 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 01:34:40.367544 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":460,"threads_started":1}
I20260925 01:34:40.368566 16863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:40.370075 16863 log.cc:824] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:40.372869 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:40.373400 16863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d: No bootstrap required, opened a new log
I20260925 01:34:40.384130 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:40.377316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32991 (local address 127.0.0.1:35463)
0925 01:34:40.377479 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:40.377484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:40.381692 (+ 4208us) server_negotiation.cc:407] Connection header received
0925 01:34:40.381931 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:40.381935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:40.381991 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:40.382117 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:40.383276 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:40.383282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:40.383285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:40.383308 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:40.383361 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:40.383381 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:40.383573 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:40.383654 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 01:34:40.383734 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:34:40.384243 16863 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } }
I20260925 01:34:40.384757 16863 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:40.384860 16863 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bec1ebd0914cd3b61b78f36cf3ec9d, State: Initialized, Role: FOLLOWER
I20260925 01:34:40.385548 16863 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } }
I20260925 01:34:40.385846 16863 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:40.386009 16863 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:40.386220 16863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:40.388526 16863 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } }
I20260925 01:34:40.389071 16863 leader_election.cc:302] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8bec1ebd0914cd3b61b78f36cf3ec9d; no voters:
I20260925 01:34:40.389737 16863 leader_election.cc:288] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:40.389882 16873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:40.390636 16873 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [term 1 LEADER]: Becoming Leader. State: Replica: e8bec1ebd0914cd3b61b78f36cf3ec9d, State: Running, Role: LEADER
I20260925 01:34:40.391227 16873 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } }
I20260925 01:34:40.392026 16863 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:40.395962 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8bec1ebd0914cd3b61b78f36cf3ec9d. Latest consensus state: current_term: 1 leader_uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } } }
I20260925 01:34:40.395929 16874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8bec1ebd0914cd3b61b78f36cf3ec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35463 } } }
I20260925 01:34:40.396446 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:40.396458 16874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:40.400081 16883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:40.405954 16883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:40.415825 16883 catalog_manager.cc:1347] Generated new cluster ID: 8701780ad29745f397faecb015865330
I20260925 01:34:40.415966 16883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:40.458503 16883 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:40.458966 16883 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:40.476559 16883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8bec1ebd0914cd3b61b78f36cf3ec9d: Generated new TSK 0
I20260925 01:34:40.477545 16883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:40.484540 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16795
W20260925 01:34:40.485666 16858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56185
W20260925 01:34:40.486418 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16795: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:40.486563 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 16795
2026-09-25T01:34:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:40 dist-test-slave-h6hf krb5kdc[16894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:40 dist-test-slave-h6hf krb5kdc[16894](info): set up 2 sockets
Sep 25 01:34:40 dist-test-slave-h6hf krb5kdc[16894](info): commencing operation
krb5kdc: starting...
W20260925 01:34:41.606527 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:41 dist-test-slave-h6hf krb5kdc[16894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:41.884191 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:42.247313 16907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:42.247710 16907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:42.247880 16907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:42.255622 16907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:42.255803 16907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:42.255935 16907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:42.256062 16907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:42.256140 16907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:42.266700 16907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:42.268167 16907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:42.269673 16907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:42.282577 16913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:42.282596 16915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:42.283195 16912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:42.283756 16907 server_base.cc:1034] running on GCE node
I20260925 01:34:42.284603 16907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:42.286303 16907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:42.287627 16907 hybrid_clock.cc:648] HybridClock initialized: now 1790300082287556 us; error 56 us; skew 500 ppm
I20260925 01:34:42.288151 16907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:42 dist-test-slave-h6hf krb5kdc[16894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:42.298141 16907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:42.299824 16907 webserver.cc:460] Webserver started at http://127.0.0.1:46431/ using document root <none> and password file <none>
I20260925 01:34:42.300562 16907 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:42.300673 16907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:42.300947 16907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:42.304140 16907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2"
format_stamp: "Formatted at 2026-09-25 01:34:42 on dist-test-slave-h6hf"
server_key: "bdb78cc9e5dee1aa597e6010974401da"
server_key_iv: "eca1379d8dad41833dbab036f8fa5a74"
server_key_version: "encryptionkey@0"
I20260925 01:34:42.304891 16907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2"
format_stamp: "Formatted at 2026-09-25 01:34:42 on dist-test-slave-h6hf"
server_key: "bdb78cc9e5dee1aa597e6010974401da"
server_key_iv: "eca1379d8dad41833dbab036f8fa5a74"
server_key_version: "encryptionkey@0"
I20260925 01:34:42.310249 16907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:34:42.314029 16922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:42.315613 16907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:42.315891 16907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2"
format_stamp: "Formatted at 2026-09-25 01:34:42 on dist-test-slave-h6hf"
server_key: "bdb78cc9e5dee1aa597e6010974401da"
server_key_iv: "eca1379d8dad41833dbab036f8fa5a74"
server_key_version: "encryptionkey@0"
I20260925 01:34:42.316085 16907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:42.335180 16907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:42.339897 16907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:42.340219 16907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:42.359669 16907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42419
I20260925 01:34:42.359727 16973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42419 every 8 connection(s)
I20260925 01:34:42.361187 16907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:42.361896 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 16907
I20260925 01:34:42.362093 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:42.362680 28288 external_mini_cluster.cc:1428] Setting key 979da6e3cff4cb8073544a3abd6e2bf0
I20260925 01:34:42.365906 16974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:34:42 dist-test-slave-h6hf krb5kdc[16894](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:34:42.380048 16974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2: Bootstrap starting.
I20260925 01:34:42.383420 16974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:42.384527 16974 log.cc:824] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:42.387441 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:42.364697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:43901 (local address 127.0.0.1:42419)
0925 01:34:42.365519 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:42.365532 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:42.365574 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:34:42.367267 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:42.367290 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:42.367718 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:42.368217 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:42.370348 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:42.371759 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:42.373743 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:42.374446 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:42.374584 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:42.379660 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:42.379668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:42.379685 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:42.379756 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:42.382987 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:42.383894 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:42.383901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:42.383907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:42.384078 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:42.384584 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:42.384588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:42.384590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:42.385276 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:42.385626 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:42.386135 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 01:34:42.386464 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":403,"threads_started":1}
I20260925 01:34:42.387732 16974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2: No bootstrap required, opened a new log
I20260925 01:34:42.391641 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:42.396727 16974 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260925 01:34:42.397156 16974 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:42.397285 16974 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3f4da62ffb4d1f8d6cf104bb7af5e2, State: Initialized, Role: FOLLOWER
I20260925 01:34:42.397837 16974 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260925 01:34:42.398078 16974 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:42.398248 16974 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:42.398433 16974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:42.400699 16974 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260925 01:34:42.401314 16974 leader_election.cc:302] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb3f4da62ffb4d1f8d6cf104bb7af5e2; no voters:
I20260925 01:34:42.402000 16974 leader_election.cc:288] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:42.402104 16984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:42.404675 16974 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:42.404277 16984 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [term 1 LEADER]: Becoming Leader. State: Replica: cb3f4da62ffb4d1f8d6cf104bb7af5e2, State: Running, Role: LEADER
I20260925 01:34:42.405633 16984 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } }
I20260925 01:34:42.409474 16985 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } } }
I20260925 01:34:42.409904 16985 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:42.413928 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:42.395921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:50061 (local address 127.0.0.1:42419)
0925 01:34:42.396074 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:42.396078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:42.396583 (+ 505us) server_negotiation.cc:407] Connection header received
0925 01:34:42.396802 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:42.396807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:42.396879 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:42.396986 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:42.398493 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:42.399328 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:42.400541 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:42.401037 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:42.401197 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:42.404059 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:42.404068 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:42.404078 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:42.404145 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:42.408100 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:42.409398 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:42.409406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:42.409413 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:42.409754 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:42.410505 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:42.410513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:42.410520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:42.411022 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:42.411392 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:42.413148 (+ 1756us) server_negotiation.cc:299] Negotiation successful
0925 01:34:42.413422 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:34:42.409507 16986 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb3f4da62ffb4d1f8d6cf104bb7af5e2. Latest consensus state: current_term: 1 leader_uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3f4da62ffb4d1f8d6cf104bb7af5e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42419 } } }
I20260925 01:34:42.416404 16986 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:42.423278 16992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:42.429090 16992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:42.439194 16992 catalog_manager.cc:1347] Generated new cluster ID: f36960d083354790a9c36807ce7e0f90
I20260925 01:34:42.439342 16992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:42.460810 16992 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:42.462809 16992 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:42.503206 16992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb3f4da62ffb4d1f8d6cf104bb7af5e2: Generated new TSK 0
I20260925 01:34:42.504187 16992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:42.519980 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 16907
I20260925 01:34:42.521386 16969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:42.618126 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 16907
W20260925 01:34:42.635761 16892 connection.cc:535] client connection to 127.0.0.1:42419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42419): Connection reset by peer (error 104)
2026-09-25T01:34:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:42 dist-test-slave-h6hf krb5kdc[17006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:42 dist-test-slave-h6hf krb5kdc[17006](info): set up 2 sockets
Sep 25 01:34:42 dist-test-slave-h6hf krb5kdc[17006](info): commencing operation
krb5kdc: starting...
W20260925 01:34:43.758574 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:43 dist-test-slave-h6hf krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:44.048026 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:44.389516 17019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:44.389818 17019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:44.389911 17019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:44.397473 17019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:44.397598 17019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:44.397645 17019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:44.397706 17019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:44.397743 17019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:44.407793 17019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:44.409221 17019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:44.410579 17019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:44.423075 17024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:44.423007 17027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:44.423043 17025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:44.424032 17019 server_base.cc:1034] running on GCE node
I20260925 01:34:44.424818 17019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:44.426326 17019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:44.427532 17019 hybrid_clock.cc:648] HybridClock initialized: now 1790300084427467 us; error 57 us; skew 500 ppm
I20260925 01:34:44.427913 17019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:44 dist-test-slave-h6hf krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:44.431278 17019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:44.432813 17019 webserver.cc:460] Webserver started at http://127.0.0.1:37563/ using document root <none> and password file <none>
I20260925 01:34:44.433575 17019 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:44.433694 17019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:44.433947 17019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:44.436981 17019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "73234509d5514be1a80ed2c8f878786b"
format_stamp: "Formatted at 2026-09-25 01:34:44 on dist-test-slave-h6hf"
server_key: "476115ab869e87a4d2f648bb0503070b"
server_key_iv: "9e563671e5370309920e47fc455e83df"
server_key_version: "encryptionkey@0"
I20260925 01:34:44.437830 17019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "73234509d5514be1a80ed2c8f878786b"
format_stamp: "Formatted at 2026-09-25 01:34:44 on dist-test-slave-h6hf"
server_key: "476115ab869e87a4d2f648bb0503070b"
server_key_iv: "9e563671e5370309920e47fc455e83df"
server_key_version: "encryptionkey@0"
I20260925 01:34:44.443303 17019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:34:44.446933 17034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:44.448491 17019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:34:44.448688 17019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "73234509d5514be1a80ed2c8f878786b"
format_stamp: "Formatted at 2026-09-25 01:34:44 on dist-test-slave-h6hf"
server_key: "476115ab869e87a4d2f648bb0503070b"
server_key_iv: "9e563671e5370309920e47fc455e83df"
server_key_version: "encryptionkey@0"
I20260925 01:34:44.448937 17019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:44.466285 17019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:44.470896 17019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:44.471203 17019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:44.490790 17019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37801
I20260925 01:34:44.490844 17085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37801 every 8 connection(s)
I20260925 01:34:44.492273 17019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:44.495393 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17019
I20260925 01:34:44.495587 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:44.496068 28288 external_mini_cluster.cc:1428] Setting key 6d4b3f81acb4ad8ef8dc62912f292d21
I20260925 01:34:44.496431 17086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:44.512868 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:44.498101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:47923 (local address 127.0.0.1:37801)
0925 01:34:44.498954 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:44.498970 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:44.499040 (+ 70us) server_negotiation.cc:407] Connection header received
0925 01:34:44.501536 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:44.501578 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:44.502181 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:44.502750 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:44.503377 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:44.505534 (+ 2157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:44.507095 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:44.508241 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:44.508448 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:44.509810 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:44.509826 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:44.509848 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:44.510667 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:44.510767 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:44.510810 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:44.510975 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:44.511244 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 01:34:44.511439 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":335,"threads_started":1}
I20260925 01:34:44.513046 17086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b: Bootstrap starting.
I20260925 01:34:44.516933 17086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:44.517220 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:44.518193 17086 log.cc:824] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:44.521588 17086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b: No bootstrap required, opened a new log
I20260925 01:34:44.529093 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:44.522095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:34811 (local address 127.0.0.1:37801)
0925 01:34:44.522275 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:44.522279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:44.522839 (+ 560us) server_negotiation.cc:407] Connection header received
0925 01:34:44.523074 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:44.523080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:44.523165 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:44.523272 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:44.523853 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:44.524676 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:44.526161 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:44.526689 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:44.526849 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:44.528248 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:44.528256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:44.528260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:44.528284 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:44.528335 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:44.528355 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:44.528500 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:44.528696 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 01:34:44.528776 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:34:44.532002 17086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73234509d5514be1a80ed2c8f878786b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37801 } }
I20260925 01:34:44.532475 17086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:44.532610 17086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73234509d5514be1a80ed2c8f878786b, State: Initialized, Role: FOLLOWER
I20260925 01:34:44.533339 17086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73234509d5514be1a80ed2c8f878786b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37801 } }
I20260925 01:34:44.533668 17086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:44.533850 17086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:44.534044 17086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:44.537314 17086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73234509d5514be1a80ed2c8f878786b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37801 } }
I20260925 01:34:44.537956 17086 leader_election.cc:302] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73234509d5514be1a80ed2c8f878786b; no voters:
I20260925 01:34:44.538594 17086 leader_election.cc:288] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:44.538710 17096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:44.539603 17096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [term 1 LEADER]: Becoming Leader. State: Replica: 73234509d5514be1a80ed2c8f878786b, State: Running, Role: LEADER
I20260925 01:34:44.540223 17096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73234509d5514be1a80ed2c8f878786b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37801 } }
I20260925 01:34:44.541141 17086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:44.543438 17097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73234509d5514be1a80ed2c8f878786b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73234509d5514be1a80ed2c8f878786b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37801 } } }
I20260925 01:34:44.543464 17098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73234509d5514be1a80ed2c8f878786b. Latest consensus state: current_term: 1 leader_uuid: "73234509d5514be1a80ed2c8f878786b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73234509d5514be1a80ed2c8f878786b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37801 } } }
I20260925 01:34:44.544036 17097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:44.544127 17098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:44.549489 17103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:44.556013 17103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:44.566627 17103 catalog_manager.cc:1347] Generated new cluster ID: 67e355333045444282e813dae5466ef3
I20260925 01:34:44.566773 17103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:44.580900 17103 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:44.582567 17103 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:44.595111 17103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73234509d5514be1a80ed2c8f878786b: Generated new TSK 0
I20260925 01:34:44.596027 17103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:44.605631 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17019
W20260925 01:34:44.607692 17081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.201:47923
W20260925 01:34:44.608498 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17019: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:44.608721 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17019
W20260925 01:34:44.625918 17004 connection.cc:535] client connection to 127.0.0.1:37801 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37801): Connection reset by peer (error 104)
2026-09-25T01:34:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:44 dist-test-slave-h6hf krb5kdc[17117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:44 dist-test-slave-h6hf krb5kdc[17117](info): set up 2 sockets
Sep 25 01:34:44 dist-test-slave-h6hf krb5kdc[17117](info): commencing operation
krb5kdc: starting...
W20260925 01:34:45.802788 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:45 dist-test-slave-h6hf krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:46.084678 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:46.429025 17130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:46.429417 17130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:46.429569 17130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:46.437037 17130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:46.437222 17130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:46.437284 17130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:46.437346 17130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:46.437393 17130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:46.447751 17130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:46.449069 17130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:46.450533 17130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:46.462945 17135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:46.463037 17138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:46.463007 17136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:46.463951 17130 server_base.cc:1034] running on GCE node
I20260925 01:34:46.464884 17130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:46.466477 17130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:46.467728 17130 hybrid_clock.cc:648] HybridClock initialized: now 1790300086467679 us; error 45 us; skew 500 ppm
I20260925 01:34:46.468103 17130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:46 dist-test-slave-h6hf krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:46.471437 17130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:46.473085 17130 webserver.cc:460] Webserver started at http://127.0.0.1:46489/ using document root <none> and password file <none>
I20260925 01:34:46.473847 17130 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:46.473995 17130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:46.474318 17130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:46.477451 17130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "06e1a46e411a4c48a0249c85284559d9"
format_stamp: "Formatted at 2026-09-25 01:34:46 on dist-test-slave-h6hf"
server_key: "e9b5c892e755894b91512e7804da1db9"
server_key_iv: "485a1de6a14540e7c22c11c41e3bcd2b"
server_key_version: "encryptionkey@0"
I20260925 01:34:46.478278 17130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "06e1a46e411a4c48a0249c85284559d9"
format_stamp: "Formatted at 2026-09-25 01:34:46 on dist-test-slave-h6hf"
server_key: "e9b5c892e755894b91512e7804da1db9"
server_key_iv: "485a1de6a14540e7c22c11c41e3bcd2b"
server_key_version: "encryptionkey@0"
I20260925 01:34:46.483559 17130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 01:34:46.487140 17145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:46.488605 17130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:34:46.488801 17130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "06e1a46e411a4c48a0249c85284559d9"
format_stamp: "Formatted at 2026-09-25 01:34:46 on dist-test-slave-h6hf"
server_key: "e9b5c892e755894b91512e7804da1db9"
server_key_iv: "485a1de6a14540e7c22c11c41e3bcd2b"
server_key_version: "encryptionkey@0"
I20260925 01:34:46.489014 17130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:46.494959 17130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:46.499444 17130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:46.499823 17130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:46.519326 17130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39993
I20260925 01:34:46.519378 17196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39993 every 8 connection(s)
I20260925 01:34:46.520879 17130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:46.521602 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17130
I20260925 01:34:46.521858 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:46.522466 28288 external_mini_cluster.cc:1428] Setting key c39fe2b8cd7fa361bb7b04522ef03793
I20260925 01:34:46.526083 17197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:46.530889 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:46.524404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:40155 (local address 127.0.0.1:39993)
0925 01:34:46.525294 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:46.525307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:46.525353 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:34:46.527102 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:46.527128 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:46.527509 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:46.527861 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:46.529073 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:46.529080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:46.529098 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:46.529114 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:46.529274 (+ 160us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:46.529299 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:46.529552 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:46.529709 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 01:34:46.529921 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":377,"threads_started":1}
I20260925 01:34:46.534816 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:46.538404 17197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9: Bootstrap starting.
I20260925 01:34:46.542325 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:46.539515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:41311 (local address 127.0.0.1:39993)
0925 01:34:46.539756 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:46.539762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:46.540108 (+ 346us) server_negotiation.cc:407] Connection header received
0925 01:34:46.540334 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:46.540338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:46.540391 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:46.540480 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:46.541614 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:46.541619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:46.541622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:46.541638 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:46.541683 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:46.541706 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:46.541798 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:46.541944 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 01:34:46.541998 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 01:34:46.542693 17197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:46.544198 17197 log.cc:824] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:46.547546 17197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9: No bootstrap required, opened a new log
I20260925 01:34:46.555481 17197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e1a46e411a4c48a0249c85284559d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 01:34:46.555948 17197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:46.556078 17197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e1a46e411a4c48a0249c85284559d9, State: Initialized, Role: FOLLOWER
I20260925 01:34:46.556790 17197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e1a46e411a4c48a0249c85284559d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 01:34:46.557024 17197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:46.557206 17197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:46.557395 17197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:46.559823 17197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e1a46e411a4c48a0249c85284559d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 01:34:46.560427 17197 leader_election.cc:302] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e1a46e411a4c48a0249c85284559d9; no voters:
I20260925 01:34:46.561025 17197 leader_election.cc:288] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:46.561224 17207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:46.561880 17207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [term 1 LEADER]: Becoming Leader. State: Replica: 06e1a46e411a4c48a0249c85284559d9, State: Running, Role: LEADER
I20260925 01:34:46.562490 17207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e1a46e411a4c48a0249c85284559d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } }
I20260925 01:34:46.563225 17197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:46.565604 17208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e1a46e411a4c48a0249c85284559d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e1a46e411a4c48a0249c85284559d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260925 01:34:46.566102 17208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:46.566258 17209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e1a46e411a4c48a0249c85284559d9. Latest consensus state: current_term: 1 leader_uuid: "06e1a46e411a4c48a0249c85284559d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e1a46e411a4c48a0249c85284559d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39993 } } }
I20260925 01:34:46.566650 17209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:46.574352 17216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:46.578840 17216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:46.587522 17216 catalog_manager.cc:1347] Generated new cluster ID: 3d552ce8749e437da0099047daca9ae4
I20260925 01:34:46.587713 17216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:46.606644 17216 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:46.607196 17216 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:46.614880 17216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e1a46e411a4c48a0249c85284559d9: Generated new TSK 0
I20260925 01:34:46.615844 17216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:46.631227 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17130
W20260925 01:34:46.632511 17192 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.201:40155
W20260925 01:34:46.633325 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17130: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:46.633502 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17130
2026-09-25T01:34:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2025 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:46 dist-test-slave-h6hf krb5kdc[17228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:46 dist-test-slave-h6hf krb5kdc[17228](info): set up 2 sockets
Sep 25 01:34:46 dist-test-slave-h6hf krb5kdc[17228](info): commencing operation
krb5kdc: starting...
W20260925 01:34:47.754870 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:47 dist-test-slave-h6hf krb5kdc[17228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:48.032953 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:48.374262 17241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:48.374588 17241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:48.374750 17241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:48.382253 17241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:48.382385 17241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:48.382483 17241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:48.382592 17241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:48.382655 17241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:48.392778 17241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:48.394228 17241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:48.395608 17241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:48.408062 17247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:48.408222 17249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:48.408090 17246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:48.409365 17241 server_base.cc:1034] running on GCE node
I20260925 01:34:48.410218 17241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:48.411751 17241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:48.413010 17241 hybrid_clock.cc:648] HybridClock initialized: now 1790300088412950 us; error 53 us; skew 500 ppm
I20260925 01:34:48.413437 17241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:48 dist-test-slave-h6hf krb5kdc[17228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:48.421376 17241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:48.423143 17241 webserver.cc:460] Webserver started at http://127.0.0.1:42035/ using document root <none> and password file <none>
I20260925 01:34:48.423925 17241 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:48.424083 17241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:48.424410 17241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:48.427627 17241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "43b0b0203b3b49bdae804cefe8ab1b11"
format_stamp: "Formatted at 2026-09-25 01:34:48 on dist-test-slave-h6hf"
server_key: "4901fc2ffe548bae736762303cea691c"
server_key_iv: "3f8441775d37c05d929d1678f600f199"
server_key_version: "encryptionkey@0"
I20260925 01:34:48.428457 17241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "43b0b0203b3b49bdae804cefe8ab1b11"
format_stamp: "Formatted at 2026-09-25 01:34:48 on dist-test-slave-h6hf"
server_key: "4901fc2ffe548bae736762303cea691c"
server_key_iv: "3f8441775d37c05d929d1678f600f199"
server_key_version: "encryptionkey@0"
I20260925 01:34:48.433980 17241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:34:48.437615 17256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:48.439105 17241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:34:48.439303 17241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "43b0b0203b3b49bdae804cefe8ab1b11"
format_stamp: "Formatted at 2026-09-25 01:34:48 on dist-test-slave-h6hf"
server_key: "4901fc2ffe548bae736762303cea691c"
server_key_iv: "3f8441775d37c05d929d1678f600f199"
server_key_version: "encryptionkey@0"
I20260925 01:34:48.439520 17241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:48.455916 17241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:48.460229 17241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:48.460537 17241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:48.479655 17241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45011
I20260925 01:34:48.479686 17309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45011 every 8 connection(s)
I20260925 01:34:48.481170 17241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:48.484949 17310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:48.489616 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17241
I20260925 01:34:48.489826 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:48.490424 28288 external_mini_cluster.cc:1428] Setting key 632bd605d47ea184594d481a16c04336
I20260925 01:34:48.496996 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11: Bootstrap starting.
I20260925 01:34:48.500855 17310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:48.502115 17310 log.cc:824] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:48.505999 17310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11: No bootstrap required, opened a new log
Sep 25 01:34:48 dist-test-slave-h6hf krb5kdc[17228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:34:48.516896 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:48.492414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:46937 (local address 127.0.0.1:45011)
0925 01:34:48.493252 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:48.493278 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:48.493324 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:34:48.495360 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:48.495385 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:48.495826 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:48.496156 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:48.498061 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:48.499385 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:48.501251 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:48.501952 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:48.502088 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:48.507408 (+ 5320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:48.507423 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:48.507445 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:48.507504 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:48.511728 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:48.512574 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:48.512584 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:48.512594 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:48.512824 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:48.513346 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:48.513351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:48.513354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:48.514152 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:48.514497 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:48.514927 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 01:34:48.515353 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":344,"threads_started":1}
I20260925 01:34:48.517565 17310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b0b0203b3b49bdae804cefe8ab1b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } }
I20260925 01:34:48.518131 17310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:48.518328 17310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b0b0203b3b49bdae804cefe8ab1b11, State: Initialized, Role: FOLLOWER
I20260925 01:34:48.519042 17310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b0b0203b3b49bdae804cefe8ab1b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } }
I20260925 01:34:48.519363 17310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:48.519552 17310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:48.519769 17310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:48.521469 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:48.523337 17310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b0b0203b3b49bdae804cefe8ab1b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } }
I20260925 01:34:48.524031 17310 leader_election.cc:302] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43b0b0203b3b49bdae804cefe8ab1b11; no voters:
I20260925 01:34:48.524842 17310 leader_election.cc:288] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:48.524935 17315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:48.527261 17315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [term 1 LEADER]: Becoming Leader. State: Replica: 43b0b0203b3b49bdae804cefe8ab1b11, State: Running, Role: LEADER
I20260925 01:34:48.527958 17315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b0b0203b3b49bdae804cefe8ab1b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } }
I20260925 01:34:48.528277 17310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:48.531582 17317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43b0b0203b3b49bdae804cefe8ab1b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b0b0203b3b49bdae804cefe8ab1b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } } }
I20260925 01:34:48.531929 17318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43b0b0203b3b49bdae804cefe8ab1b11. Latest consensus state: current_term: 1 leader_uuid: "43b0b0203b3b49bdae804cefe8ab1b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43b0b0203b3b49bdae804cefe8ab1b11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45011 } } }
I20260925 01:34:48.532224 17317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:48.532297 17318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:48.536994 17325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:48.544896 17325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:48.564093 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:48.529606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:38943 (local address 127.0.0.1:45011)
0925 01:34:48.529874 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:48.529884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:48.540394 (+ 10510us) server_negotiation.cc:407] Connection header received
0925 01:34:48.540602 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:48.540607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:48.540735 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:48.540858 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:48.543331 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:48.544919 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:48.547346 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:48.548081 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:48.548246 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:48.552413 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:48.552420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:48.552425 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:48.552476 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:48.557671 (+ 5195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:48.559240 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:48.559250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:48.559257 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:48.559475 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:48.560328 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:48.560337 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:48.560344 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:48.560935 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:48.561348 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:48.563165 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0925 01:34:48.563463 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:34:48.567622 17325 catalog_manager.cc:1347] Generated new cluster ID: f9be162c560b4426b6ca0e0046f9a90a
I20260925 01:34:48.567777 17325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:48.586756 17325 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:48.588508 17325 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:48.599694 17325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43b0b0203b3b49bdae804cefe8ab1b11: Generated new TSK 0
I20260925 01:34:48.600661 17325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:48.612465 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17241
I20260925 01:34:48.614331 17305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:48.708165 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17241
W20260925 01:34:48.726117 17226 connection.cc:535] client connection to 127.0.0.1:45011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45011): Connection reset by peer (error 104)
2026-09-25T01:34:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2078 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:48 dist-test-slave-h6hf krb5kdc[17342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:48 dist-test-slave-h6hf krb5kdc[17342](info): set up 2 sockets
Sep 25 01:34:48 dist-test-slave-h6hf krb5kdc[17342](info): commencing operation
krb5kdc: starting...
W20260925 01:34:49.834511 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:50 dist-test-slave-h6hf krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:50.108304 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:50.449080 17355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:50.449446 17355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:50.449594 17355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:50.457047 17355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:50.457245 17355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:50.457350 17355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:50.457456 17355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:50.457525 17355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:50.468096 17355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:50.469698 17355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:50.471102 17355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:50.483311 17360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:50.483217 17363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:50.483346 17361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:50.484095 17355 server_base.cc:1034] running on GCE node
I20260925 01:34:50.484918 17355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:50.486456 17355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:50.487694 17355 hybrid_clock.cc:648] HybridClock initialized: now 1790300090487631 us; error 58 us; skew 500 ppm
I20260925 01:34:50.488049 17355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:50 dist-test-slave-h6hf krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:50.491158 17355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:50.492762 17355 webserver.cc:460] Webserver started at http://127.0.0.1:44195/ using document root <none> and password file <none>
I20260925 01:34:50.493557 17355 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:50.493706 17355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:50.494014 17355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:50.497293 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "ed76756e69ee484fa4870bdb3a69389a"
format_stamp: "Formatted at 2026-09-25 01:34:50 on dist-test-slave-h6hf"
server_key: "b73dcf749467411b3c4ab12fe72266c4"
server_key_iv: "8e2d03b0c169caa561e99c7933d7c708"
server_key_version: "encryptionkey@0"
I20260925 01:34:50.498143 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "ed76756e69ee484fa4870bdb3a69389a"
format_stamp: "Formatted at 2026-09-25 01:34:50 on dist-test-slave-h6hf"
server_key: "b73dcf749467411b3c4ab12fe72266c4"
server_key_iv: "8e2d03b0c169caa561e99c7933d7c708"
server_key_version: "encryptionkey@0"
I20260925 01:34:50.503794 17355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 01:34:50.507671 17371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:50.509225 17355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:34:50.509431 17355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "ed76756e69ee484fa4870bdb3a69389a"
format_stamp: "Formatted at 2026-09-25 01:34:50 on dist-test-slave-h6hf"
server_key: "b73dcf749467411b3c4ab12fe72266c4"
server_key_iv: "8e2d03b0c169caa561e99c7933d7c708"
server_key_version: "encryptionkey@0"
I20260925 01:34:50.509652 17355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:50.534387 17355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:50.539111 17355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:50.539440 17355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:50.559824 17355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33939
I20260925 01:34:50.559854 17422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33939 every 8 connection(s)
I20260925 01:34:50.561440 17355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:50.565440 17423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:50.566306 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17355
I20260925 01:34:50.566495 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:50.567095 28288 external_mini_cluster.cc:1428] Setting key 9d17e55ebe4d6b3116609b05cd084cee
I20260925 01:34:50.580447 17423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a: Bootstrap starting.
I20260925 01:34:50.581174 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:50.568874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:34275 (local address 127.0.0.1:33939)
0925 01:34:50.569735 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:50.569748 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:50.569792 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:34:50.571318 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:50.571341 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:50.571771 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:50.572151 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:50.572757 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:50.574205 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:50.576111 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:50.576947 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:50.577100 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:50.578483 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:50.578491 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:50.578507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:50.579127 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:50.579211 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:50.579244 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:50.579378 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:50.579792 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 01:34:50.579993 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":399,"threads_started":1}
I20260925 01:34:50.584146 17423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:50.585343 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:50.585441 17423 log.cc:824] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:50.588411 17423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a: No bootstrap required, opened a new log
I20260925 01:34:50.596362 17423 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed76756e69ee484fa4870bdb3a69389a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } }
I20260925 01:34:50.596957 17423 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:50.597043 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:50.590356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:43503 (local address 127.0.0.1:33939)
0925 01:34:50.590617 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:50.590626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:50.591181 (+ 555us) server_negotiation.cc:407] Connection header received
0925 01:34:50.591397 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:50.591402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:50.591485 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:50.591621 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:50.592146 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:50.592945 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:50.594339 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:50.594793 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:50.594972 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:50.596224 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:50.596230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:50.596234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:50.596253 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:50.596291 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:50.596308 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:50.596439 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:50.596647 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 01:34:50.596705 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:34:50.597155 17423 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed76756e69ee484fa4870bdb3a69389a, State: Initialized, Role: FOLLOWER
I20260925 01:34:50.597955 17423 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed76756e69ee484fa4870bdb3a69389a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } }
I20260925 01:34:50.598245 17423 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:50.598426 17423 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:50.598714 17423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:50.601291 17423 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed76756e69ee484fa4870bdb3a69389a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } }
I20260925 01:34:50.601816 17423 leader_election.cc:302] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed76756e69ee484fa4870bdb3a69389a; no voters:
I20260925 01:34:50.602438 17423 leader_election.cc:288] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:50.602640 17433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:50.603503 17433 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [term 1 LEADER]: Becoming Leader. State: Replica: ed76756e69ee484fa4870bdb3a69389a, State: Running, Role: LEADER
I20260925 01:34:50.604135 17433 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed76756e69ee484fa4870bdb3a69389a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } }
I20260925 01:34:50.604866 17423 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:50.607539 17435 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed76756e69ee484fa4870bdb3a69389a. Latest consensus state: current_term: 1 leader_uuid: "ed76756e69ee484fa4870bdb3a69389a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed76756e69ee484fa4870bdb3a69389a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } } }
I20260925 01:34:50.607950 17435 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:50.607719 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed76756e69ee484fa4870bdb3a69389a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed76756e69ee484fa4870bdb3a69389a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33939 } } }
I20260925 01:34:50.608134 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:50.613983 17441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:50.620734 17441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:50.631740 17441 catalog_manager.cc:1347] Generated new cluster ID: 7f626cb722544109945203c24179271e
I20260925 01:34:50.631875 17441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:50.646582 17441 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:50.648617 17441 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:50.662024 17441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed76756e69ee484fa4870bdb3a69389a: Generated new TSK 0
I20260925 01:34:50.663384 17441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:50.675283 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17355
W20260925 01:34:50.677384 17418 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.201:34275
W20260925 01:34:50.678320 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17355: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:50.678490 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17355
W20260925 01:34:50.697326 17340 connection.cc:535] client connection to 127.0.0.1:33939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33939): Connection reset by peer (error 104)
2026-09-25T01:34:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:50 dist-test-slave-h6hf krb5kdc[17454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:50 dist-test-slave-h6hf krb5kdc[17454](info): set up 2 sockets
Sep 25 01:34:50 dist-test-slave-h6hf krb5kdc[17454](info): commencing operation
krb5kdc: starting...
W20260925 01:34:51.790591 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:51 dist-test-slave-h6hf krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:34:52.069883 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:52.429356 17467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:52.429668 17467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:52.429862 17467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:52.437414 17467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:52.437526 17467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:52.437633 17467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:52.437738 17467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:52.437824 17467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:52.447947 17467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:52.449352 17467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:52.450724 17467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:52.463451 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:52.463433 17473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:52.463507 17475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:52.464679 17467 server_base.cc:1034] running on GCE node
I20260925 01:34:52.465595 17467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:52.467096 17467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:52.468357 17467 hybrid_clock.cc:648] HybridClock initialized: now 1790300092468296 us; error 55 us; skew 500 ppm
I20260925 01:34:52.468756 17467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:52 dist-test-slave-h6hf krb5kdc[17454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:52.471988 17467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:34:52.473635 17467 webserver.cc:460] Webserver started at http://127.0.0.1:45243/ using document root <none> and password file <none>
I20260925 01:34:52.474351 17467 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:52.474499 17467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:52.474807 17467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:52.477885 17467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "61e5b347be5f49b78d53c7587f16f269"
format_stamp: "Formatted at 2026-09-25 01:34:52 on dist-test-slave-h6hf"
server_key: "fb108c4e343932a7c54cb05483313222"
server_key_iv: "edd64a170a4d5f27f9a76ede0e475896"
server_key_version: "encryptionkey@0"
I20260925 01:34:52.478677 17467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "61e5b347be5f49b78d53c7587f16f269"
format_stamp: "Formatted at 2026-09-25 01:34:52 on dist-test-slave-h6hf"
server_key: "fb108c4e343932a7c54cb05483313222"
server_key_iv: "edd64a170a4d5f27f9a76ede0e475896"
server_key_version: "encryptionkey@0"
I20260925 01:34:52.483754 17467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:34:52.487268 17482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:52.488579 17467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:34:52.488770 17467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "61e5b347be5f49b78d53c7587f16f269"
format_stamp: "Formatted at 2026-09-25 01:34:52 on dist-test-slave-h6hf"
server_key: "fb108c4e343932a7c54cb05483313222"
server_key_iv: "edd64a170a4d5f27f9a76ede0e475896"
server_key_version: "encryptionkey@0"
I20260925 01:34:52.488974 17467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:52.494863 17467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:52.499197 17467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:52.499542 17467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:52.518774 17467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36351
I20260925 01:34:52.518821 17534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36351 every 8 connection(s)
I20260925 01:34:52.520264 17467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:52.524211 17535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:52.527859 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17467
I20260925 01:34:52.528085 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:52.528688 28288 external_mini_cluster.cc:1428] Setting key d13aa6641e13188def669a7ea91b1808
I20260925 01:34:52.537784 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:52.530945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:56275 (local address 127.0.0.1:36351)
0925 01:34:52.531878 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:52.531891 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:52.531937 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:34:52.533875 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:52.533901 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:52.534343 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:52.534744 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:52.535914 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:52.535922 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:52.535935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:52.535951 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:52.536021 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:52.536069 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:52.536276 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:52.536394 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 01:34:52.536636 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":409,"threads_started":1}
I20260925 01:34:52.538170 17535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269: Bootstrap starting.
I20260925 01:34:52.541553 17535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:52.541960 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:52.542932 17535 log.cc:824] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:52.546541 17535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269: No bootstrap required, opened a new log
I20260925 01:34:52.549845 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:52.547233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.201:34137 (local address 127.0.0.1:36351)
0925 01:34:52.547429 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:52.547436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:52.547871 (+ 435us) server_negotiation.cc:407] Connection header received
0925 01:34:52.548009 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:52.548014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:52.548076 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:52.548199 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:52.549154 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:52.549162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:52.549170 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:52.549198 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:52.549251 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:52.549271 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:52.549396 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:52.549456 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 01:34:52.549527 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:34:52.555464 17535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e5b347be5f49b78d53c7587f16f269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260925 01:34:52.555871 17535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:52.555974 17535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e5b347be5f49b78d53c7587f16f269, State: Initialized, Role: FOLLOWER
I20260925 01:34:52.556545 17535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e5b347be5f49b78d53c7587f16f269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260925 01:34:52.556790 17535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:52.556943 17535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:52.557178 17535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:52.559479 17535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e5b347be5f49b78d53c7587f16f269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260925 01:34:52.560019 17535 leader_election.cc:302] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61e5b347be5f49b78d53c7587f16f269; no voters:
I20260925 01:34:52.560704 17535 leader_election.cc:288] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:52.560899 17545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:52.561858 17545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [term 1 LEADER]: Becoming Leader. State: Replica: 61e5b347be5f49b78d53c7587f16f269, State: Running, Role: LEADER
I20260925 01:34:52.562503 17545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e5b347be5f49b78d53c7587f16f269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } }
I20260925 01:34:52.563089 17535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:52.565660 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61e5b347be5f49b78d53c7587f16f269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e5b347be5f49b78d53c7587f16f269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } } }
I20260925 01:34:52.565629 17547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61e5b347be5f49b78d53c7587f16f269. Latest consensus state: current_term: 1 leader_uuid: "61e5b347be5f49b78d53c7587f16f269" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61e5b347be5f49b78d53c7587f16f269" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36351 } } }
I20260925 01:34:52.566208 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:52.566253 17547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:52.567982 17552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:52.574379 17552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:52.584733 17552 catalog_manager.cc:1347] Generated new cluster ID: 90fb26810e05499e8c064ea620d75ff2
I20260925 01:34:52.584879 17552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:52.601750 17552 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:52.602216 17552 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:52.610847 17552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61e5b347be5f49b78d53c7587f16f269: Generated new TSK 0
I20260925 01:34:52.611707 17552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:52.625873 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17467
W20260925 01:34:52.627506 17530 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.201:56275
W20260925 01:34:52.628392 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17467: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:52.628592 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17467
2026-09-25T01:34:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1948 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:52 dist-test-slave-h6hf krb5kdc[17566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:52 dist-test-slave-h6hf krb5kdc[17566](info): set up 2 sockets
Sep 25 01:34:52 dist-test-slave-h6hf krb5kdc[17566](info): commencing operation
krb5kdc: starting...
W20260925 01:34:53.806668 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:53 dist-test-slave-h6hf krb5kdc[17566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:34:54.100899 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:42677
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:42677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:54.447104 17579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:54.447487 17579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:54.447660 17579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:54.455065 17579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:54.455251 17579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:54.455369 17579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:54.455488 17579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:54.455585 17579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:54.466225 17579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:42677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:42677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:54.467643 17579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:54.469033 17579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:54.481596 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:54.481695 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:54.481695 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:54.482381 17579 server_base.cc:1034] running on GCE node
I20260925 01:34:54.483269 17579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:54.484766 17579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:54.485991 17579 hybrid_clock.cc:648] HybridClock initialized: now 1790300094485937 us; error 49 us; skew 500 ppm
I20260925 01:34:54.486359 17579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:54 dist-test-slave-h6hf krb5kdc[17566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300094, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:54.489836 17579 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:34:54.491364 17579 webserver.cc:460] Webserver started at http://127.27.160.62:36211/ using document root <none> and password file <none>
I20260925 01:34:54.492105 17579 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:54.492254 17579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:54.492588 17579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:54.495779 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "3d3377f2b7d9409580483bfd89885891"
format_stamp: "Formatted at 2026-09-25 01:34:54 on dist-test-slave-h6hf"
server_key: "6b801ac67ccd654ce59b81fe956cf785"
server_key_iv: "7b06e8dcff247a126752a7da0448ace8"
server_key_version: "encryptionkey@0"
I20260925 01:34:54.496587 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "3d3377f2b7d9409580483bfd89885891"
format_stamp: "Formatted at 2026-09-25 01:34:54 on dist-test-slave-h6hf"
server_key: "6b801ac67ccd654ce59b81fe956cf785"
server_key_iv: "7b06e8dcff247a126752a7da0448ace8"
server_key_version: "encryptionkey@0"
I20260925 01:34:54.502365 17579 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:34:54.506304 17594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:54.507838 17579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:34:54.508054 17579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "3d3377f2b7d9409580483bfd89885891"
format_stamp: "Formatted at 2026-09-25 01:34:54 on dist-test-slave-h6hf"
server_key: "6b801ac67ccd654ce59b81fe956cf785"
server_key_iv: "7b06e8dcff247a126752a7da0448ace8"
server_key_version: "encryptionkey@0"
I20260925 01:34:54.508285 17579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:54.525210 17579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:54.530524 17579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:54.530859 17579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:54.550944 17579 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:42677
I20260925 01:34:54.550968 17646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:42677 every 8 connection(s)
I20260925 01:34:54.552506 17579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:54.556495 17647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:54.559278 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17579
I20260925 01:34:54.559461 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:54.560057 28288 external_mini_cluster.cc:1428] Setting key 41aa30ec56e74f66cfb1abd4bf46ddaf
I20260925 01:34:54.570312 17647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891: Bootstrap starting.
I20260925 01:34:54.574784 17647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891: Neither blocks nor log segments found. Creating new log.
Sep 25 01:34:54 dist-test-slave-h6hf krb5kdc[17566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:34:54.576164 17647 log.cc:824] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:54.579689 17647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891: No bootstrap required, opened a new log
I20260925 01:34:54.585390 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:54.562055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55937 (local address 127.27.160.62:42677)
0925 01:34:54.562963 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:54.562977 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:54.563019 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:34:54.564745 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:54.564769 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:54.565365 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:54.565782 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:54.567660 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:54.569084 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:54.571029 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:54.571772 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:54.571946 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:54.577437 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:54.577450 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:54.577468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:54.577512 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:54.580809 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:54.581780 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:54.581788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:54.581795 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:54.581973 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:54.582482 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:54.582485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:54.582487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:54.583050 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:54.583449 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:54.583947 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 01:34:54.584286 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":409,"threads_started":1}
I20260925 01:34:54.588019 17647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3377f2b7d9409580483bfd89885891" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42677 } }
I20260925 01:34:54.588433 17647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:54.588613 17647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3377f2b7d9409580483bfd89885891, State: Initialized, Role: FOLLOWER
I20260925 01:34:54.589222 17647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3377f2b7d9409580483bfd89885891" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42677 } }
I20260925 01:34:54.589499 17647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:54.589565 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:54.589669 17647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:54.589828 17647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:54.593084 17647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3377f2b7d9409580483bfd89885891" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42677 } }
I20260925 01:34:54.593833 17647 leader_election.cc:302] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d3377f2b7d9409580483bfd89885891; no voters:
I20260925 01:34:54.594467 17647 leader_election.cc:288] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:54.594578 17656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:54.598275 17656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [term 1 LEADER]: Becoming Leader. State: Replica: 3d3377f2b7d9409580483bfd89885891, State: Running, Role: LEADER
I20260925 01:34:54.599043 17656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3377f2b7d9409580483bfd89885891" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42677 } }
I20260925 01:34:54.599350 17647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:54.601749 17657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d3377f2b7d9409580483bfd89885891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3377f2b7d9409580483bfd89885891" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42677 } } }
I20260925 01:34:54.602136 17657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:54.603614 17658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d3377f2b7d9409580483bfd89885891. Latest consensus state: current_term: 1 leader_uuid: "3d3377f2b7d9409580483bfd89885891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3377f2b7d9409580483bfd89885891" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42677 } } }
I20260925 01:34:54.604014 17658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:54.605370 17662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:54.610915 17662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:54.621279 17650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:54.595281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46409 (local address 127.27.160.62:42677)
0925 01:34:54.595504 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:54.595513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:54.599045 (+ 3532us) server_negotiation.cc:407] Connection header received
0925 01:34:54.599353 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:54.599359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:54.599443 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:54.599532 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:54.601942 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:54.603528 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:54.606011 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:54.606742 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:54.606908 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:54.611169 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:54.611184 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:54.611192 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:34:54.611252 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:54.616509 (+ 5257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:54.617918 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:54.617927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:54.617932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:54.618127 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:34:54.618785 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:34:54.618792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:34:54.618797 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:34:54.619274 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:54.619535 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:54.620446 (+ 911us) server_negotiation.cc:299] Negotiation successful
0925 01:34:54.620696 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:34:54.621516 17662 catalog_manager.cc:1347] Generated new cluster ID: bfc8b94a61d94a74bab0e165db68ee5c
I20260925 01:34:54.621980 17662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:54.644215 17662 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:54.646196 17662 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:54.657294 17662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d3377f2b7d9409580483bfd89885891: Generated new TSK 0
I20260925 01:34:54.658443 17662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:54.673695 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17579
I20260925 01:34:54.674902 17642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:34:54.770761 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17579
W20260925 01:34:54.789686 17564 connection.cc:535] client connection to 127.27.160.62:42677 recv error: Network error: failed to read from TLS socket (remote: 127.27.160.62:42677): Connection reset by peer (error 104)
2026-09-25T01:34:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2143 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:54 dist-test-slave-h6hf krb5kdc[17679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:54 dist-test-slave-h6hf krb5kdc[17679](info): set up 2 sockets
Sep 25 01:34:54 dist-test-slave-h6hf krb5kdc[17679](info): commencing operation
krb5kdc: starting...
W20260925 01:34:55.914795 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:56 dist-test-slave-h6hf krb5kdc[17679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:34:56.193390 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:39055
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:39055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:56.535281 17692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:56.535609 17692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:56.535799 17692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:56.543217 17692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:56.543368 17692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:56.543464 17692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:56.543560 17692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:56.543601 17692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:56.554013 17692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:39055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:39055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:56.555454 17692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:56.556844 17692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:56.569623 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:56.569574 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:56.569593 17700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:56.570533 17692 server_base.cc:1034] running on GCE node
I20260925 01:34:56.571368 17692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:56.572918 17692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:56.574154 17692 hybrid_clock.cc:648] HybridClock initialized: now 1790300096574080 us; error 73 us; skew 500 ppm
I20260925 01:34:56.574568 17692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:56 dist-test-slave-h6hf krb5kdc[17679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300096, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:56.577962 17692 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:34:56.579718 17692 webserver.cc:460] Webserver started at http://127.27.160.62:35143/ using document root <none> and password file <none>
I20260925 01:34:56.580437 17692 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:56.580552 17692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:56.580811 17692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:56.583899 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "eada70919b94491f8912bd6705b4f41c"
format_stamp: "Formatted at 2026-09-25 01:34:56 on dist-test-slave-h6hf"
server_key: "ecce140939f0b05c63ae1d4ab467b026"
server_key_iv: "8fc545c1146098ce5581668412f28e5f"
server_key_version: "encryptionkey@0"
I20260925 01:34:56.584658 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "eada70919b94491f8912bd6705b4f41c"
format_stamp: "Formatted at 2026-09-25 01:34:56 on dist-test-slave-h6hf"
server_key: "ecce140939f0b05c63ae1d4ab467b026"
server_key_iv: "8fc545c1146098ce5581668412f28e5f"
server_key_version: "encryptionkey@0"
I20260925 01:34:56.590112 17692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 01:34:56.593945 17707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:56.595438 17692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:34:56.595628 17692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "eada70919b94491f8912bd6705b4f41c"
format_stamp: "Formatted at 2026-09-25 01:34:56 on dist-test-slave-h6hf"
server_key: "ecce140939f0b05c63ae1d4ab467b026"
server_key_iv: "8fc545c1146098ce5581668412f28e5f"
server_key_version: "encryptionkey@0"
I20260925 01:34:56.595831 17692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:56.613001 17692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:56.617640 17692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:56.617937 17692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:56.638069 17692 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:39055
I20260925 01:34:56.638109 17758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:39055 every 8 connection(s)
I20260925 01:34:56.639534 17692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:56.640702 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17692
I20260925 01:34:56.640913 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:56.641489 28288 external_mini_cluster.cc:1428] Setting key c6e43e2313da9a76498437609e4d9a0c
I20260925 01:34:56.644450 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:56.657030 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:56.643622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48619 (local address 127.27.160.62:39055)
0925 01:34:56.644496 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:56.644547 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:56.644605 (+ 58us) server_negotiation.cc:407] Connection header received
0925 01:34:56.646530 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:56.646560 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:56.646988 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:56.647364 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:56.648013 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:56.649742 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:56.651692 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:56.652472 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:56.652645 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:56.654205 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:56.654215 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:56.654233 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:56.654975 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:56.655081 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:56.655113 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:56.655260 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:56.655711 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 01:34:56.655934 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":412,"threads_started":1}
I20260925 01:34:56.657842 17759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c: Bootstrap starting.
I20260925 01:34:56.661363 17759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:56.661501 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:56.662680 17759 log.cc:824] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:56.666589 17759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c: No bootstrap required, opened a new log
I20260925 01:34:56.676425 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:56.668922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45519 (local address 127.27.160.62:39055)
0925 01:34:56.669196 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:56.669207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:56.669561 (+ 354us) server_negotiation.cc:407] Connection header received
0925 01:34:56.669911 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:56.669916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:56.669992 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:56.670089 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:56.670745 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:56.671639 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:56.673593 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:34:56.674100 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:34:56.674247 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:34:56.675534 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:56.675543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:56.675547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:56.675570 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:56.675616 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:56.675647 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:56.675746 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:56.675958 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 01:34:56.676010 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 01:34:56.677356 17759 raft_consensus.cc:348] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eada70919b94491f8912bd6705b4f41c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39055 } }
I20260925 01:34:56.677784 17759 raft_consensus.cc:374] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:56.677920 17759 raft_consensus.cc:729] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eada70919b94491f8912bd6705b4f41c, State: Initialized, Role: FOLLOWER
I20260925 01:34:56.678548 17759 consensus_queue.cc:260] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eada70919b94491f8912bd6705b4f41c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39055 } }
I20260925 01:34:56.678828 17759 raft_consensus.cc:388] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:56.678979 17759 raft_consensus.cc:482] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:56.679186 17759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:56.681763 17759 raft_consensus.cc:504] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eada70919b94491f8912bd6705b4f41c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39055 } }
I20260925 01:34:56.682299 17759 leader_election.cc:302] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eada70919b94491f8912bd6705b4f41c; no voters:
I20260925 01:34:56.682904 17759 leader_election.cc:288] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:56.683089 17769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:56.684433 17769 raft_consensus.cc:686] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [term 1 LEADER]: Becoming Leader. State: Replica: eada70919b94491f8912bd6705b4f41c, State: Running, Role: LEADER
I20260925 01:34:56.685042 17769 consensus_queue.cc:237] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eada70919b94491f8912bd6705b4f41c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39055 } }
I20260925 01:34:56.685261 17759 sys_catalog.cc:564] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:56.688284 17770 sys_catalog.cc:455] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eada70919b94491f8912bd6705b4f41c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eada70919b94491f8912bd6705b4f41c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39055 } } }
I20260925 01:34:56.688235 17771 sys_catalog.cc:455] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [sys.catalog]: SysCatalogTable state changed. Reason: New leader eada70919b94491f8912bd6705b4f41c. Latest consensus state: current_term: 1 leader_uuid: "eada70919b94491f8912bd6705b4f41c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eada70919b94491f8912bd6705b4f41c" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 39055 } } }
I20260925 01:34:56.688705 17771 sys_catalog.cc:458] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:56.688705 17770 sys_catalog.cc:458] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:56.689985 17778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:56.696719 17778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:56.707291 17778 catalog_manager.cc:1347] Generated new cluster ID: 136ba0b907bc499183133a350c5ec10e
I20260925 01:34:56.707435 17778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:56.721822 17778 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:56.723223 17778 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:56.732465 17778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eada70919b94491f8912bd6705b4f41c: Generated new TSK 0
I20260925 01:34:56.733522 17778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:56.742805 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17692
W20260925 01:34:56.744284 17754 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48619
W20260925 01:34:56.745210 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17692: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:56.745393 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17692
W20260925 01:34:56.760470 17677 connection.cc:535] client connection to 127.27.160.62:39055 recv error: Network error: failed to read from TLS socket (remote: 127.27.160.62:39055): Connection reset by peer (error 104)
2026-09-25T01:34:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1970 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:56 dist-test-slave-h6hf krb5kdc[17790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:56 dist-test-slave-h6hf krb5kdc[17790](info): set up 2 sockets
Sep 25 01:34:56 dist-test-slave-h6hf krb5kdc[17790](info): commencing operation
krb5kdc: starting...
W20260925 01:34:57.878777 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:34:58 dist-test-slave-h6hf krb5kdc[17790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:34:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:34:58.168546 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:45137
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:45137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:34:58.527338 17803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:34:58.527681 17803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:34:58.527882 17803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:34:58.535754 17803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:34:58.535884 17803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:34:58.535974 17803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:34:58.536064 17803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:34:58.536104 17803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:34:58.546475 17803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:45137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:45137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:34:58.547901 17803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:34:58.549386 17803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:34:58.562007 17811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:58.562086 17809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:34:58.562086 17808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:34:58.562983 17803 server_base.cc:1034] running on GCE node
I20260925 01:34:58.563880 17803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:34:58.565411 17803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:34:58.566710 17803 hybrid_clock.cc:648] HybridClock initialized: now 1790300098566655 us; error 49 us; skew 500 ppm
I20260925 01:34:58.567094 17803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:34:58 dist-test-slave-h6hf krb5kdc[17790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300098, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:34:58.570573 17803 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:34:58.572402 17803 webserver.cc:460] Webserver started at http://127.27.160.62:34401/ using document root <none> and password file <none>
I20260925 01:34:58.573297 17803 fs_manager.cc:359] Metadata directory not provided
I20260925 01:34:58.573465 17803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:34:58.573812 17803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:34:58.577310 17803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2"
format_stamp: "Formatted at 2026-09-25 01:34:58 on dist-test-slave-h6hf"
server_key: "e98d0e029567a4acddd672bc7c5f9892"
server_key_iv: "9d4f9f9dc6678ce074f008b5007b4801"
server_key_version: "encryptionkey@0"
I20260925 01:34:58.578177 17803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2"
format_stamp: "Formatted at 2026-09-25 01:34:58 on dist-test-slave-h6hf"
server_key: "e98d0e029567a4acddd672bc7c5f9892"
server_key_iv: "9d4f9f9dc6678ce074f008b5007b4801"
server_key_version: "encryptionkey@0"
I20260925 01:34:58.584197 17803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 01:34:58.588222 17818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:34:58.589905 17803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:34:58.590154 17803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2"
format_stamp: "Formatted at 2026-09-25 01:34:58 on dist-test-slave-h6hf"
server_key: "e98d0e029567a4acddd672bc7c5f9892"
server_key_iv: "9d4f9f9dc6678ce074f008b5007b4801"
server_key_version: "encryptionkey@0"
I20260925 01:34:58.590404 17803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:34:58.597107 17803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:34:58.602041 17803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:34:58.602391 17803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:34:58.624063 17803 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:45137
I20260925 01:34:58.624085 17870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:45137 every 8 connection(s)
I20260925 01:34:58.625690 17803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:34:58.627251 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17803
I20260925 01:34:58.627431 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:34:58.627967 28288 external_mini_cluster.cc:1428] Setting key c3a72428bf4d8e86f7fc58965675b2b8
I20260925 01:34:58.630596 17871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:34:58.637022 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:58.629945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50743 (local address 127.27.160.62:45137)
0925 01:34:58.630843 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:58.630856 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:58.630901 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:34:58.632741 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:58.632766 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:58.633295 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:58.633808 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:58.635104 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:58.635111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:58.635126 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:58.635141 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:58.635278 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:58.635301 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:58.635530 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:58.635647 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 01:34:58.635897 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":389,"threads_started":1}
I20260925 01:34:58.641564 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:34:58.643323 17871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2: Bootstrap starting.
I20260925 01:34:58.647550 17871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2: Neither blocks nor log segments found. Creating new log.
I20260925 01:34:58.649006 17871 log.cc:824] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2: Log is configured to *not* fsync() on all Append() calls
I20260925 01:34:58.650233 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:34:58.647109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51279 (local address 127.27.160.62:45137)
0925 01:34:58.647319 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:34:58.647327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:34:58.647814 (+ 487us) server_negotiation.cc:407] Connection header received
0925 01:34:58.648012 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:34:58.648017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:34:58.648105 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:34:58.648228 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:34:58.649362 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:34:58.649367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:34:58.649371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:34:58.649390 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:34:58.649436 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:34:58.649459 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:34:58.649604 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 01:34:58.649839 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 01:34:58.649907 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:34:58.652350 17871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2: No bootstrap required, opened a new log
I20260925 01:34:58.659861 17871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 45137 } }
I20260925 01:34:58.660271 17871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:34:58.660401 17871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ef1f3a7c4544fa82bb16c78ba8c6a2, State: Initialized, Role: FOLLOWER
I20260925 01:34:58.661044 17871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 45137 } }
I20260925 01:34:58.661338 17871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:34:58.661499 17871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:34:58.661666 17871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:34:58.664191 17871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 45137 } }
I20260925 01:34:58.664714 17871 leader_election.cc:302] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50ef1f3a7c4544fa82bb16c78ba8c6a2; no voters:
I20260925 01:34:58.665416 17871 leader_election.cc:288] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:34:58.665575 17881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:34:58.666335 17881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [term 1 LEADER]: Becoming Leader. State: Replica: 50ef1f3a7c4544fa82bb16c78ba8c6a2, State: Running, Role: LEADER
I20260925 01:34:58.666956 17881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 45137 } }
I20260925 01:34:58.667544 17871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:34:58.670817 17882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 45137 } } }
I20260925 01:34:58.671382 17882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:58.670548 17883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50ef1f3a7c4544fa82bb16c78ba8c6a2. Latest consensus state: current_term: 1 leader_uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50ef1f3a7c4544fa82bb16c78ba8c6a2" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 45137 } } }
I20260925 01:34:58.672541 17883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:34:58.673820 17890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:34:58.679157 17890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:34:58.679958 17898 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:34:58.689245 17890 catalog_manager.cc:1347] Generated new cluster ID: 892c82b632774aa9836f9a224db256dc
I20260925 01:34:58.689390 17890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:34:58.709925 17890 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:34:58.710542 17890 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:34:58.725399 17890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50ef1f3a7c4544fa82bb16c78ba8c6a2: Generated new TSK 0
I20260925 01:34:58.726605 17890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:34:58.732724 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17803
W20260925 01:34:58.734390 17866 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50743
W20260925 01:34:58.735244 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17803: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:34:58.735422 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17803
2026-09-25T01:34:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:34:58 dist-test-slave-h6hf krb5kdc[17902](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:34:58 dist-test-slave-h6hf krb5kdc[17902](info): set up 2 sockets
Sep 25 01:34:58 dist-test-slave-h6hf krb5kdc[17902](info): commencing operation
krb5kdc: starting...
W20260925 01:34:59.902627 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:35:00 dist-test-slave-h6hf krb5kdc[17902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:35:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:35:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:35:00.182701 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:43913
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:43913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:35:00.530632 17915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:00.531018 17915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:00.531222 17915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:00.538851 17915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:00.539026 17915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:00.539170 17915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:00.539294 17915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:00.539340 17915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:00.549816 17915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:43913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:43913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:00.551151 17915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:00.552448 17915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:35:00.565027 17920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:00.565079 17921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:00.565104 17923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:00.566481 17915 server_base.cc:1034] running on GCE node
I20260925 01:35:00.567406 17915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:00.568975 17915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:00.570225 17915 hybrid_clock.cc:648] HybridClock initialized: now 1790300100570183 us; error 68 us; skew 500 ppm
I20260925 01:35:00.570644 17915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:00 dist-test-slave-h6hf krb5kdc[17902](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300100, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:00.580436 17915 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:35:00.582357 17915 webserver.cc:460] Webserver started at http://127.27.160.62:44123/ using document root <none> and password file <none>
I20260925 01:35:00.583110 17915 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:00.583279 17915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:00.583595 17915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:00.586779 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "2280e05dcefd4289864773b8a130ed1d"
format_stamp: "Formatted at 2026-09-25 01:35:00 on dist-test-slave-h6hf"
server_key: "b9cb0ef837e72cc1647ccf913efba125"
server_key_iv: "99cbf3ce32ec87efdc2c50729451381a"
server_key_version: "encryptionkey@0"
I20260925 01:35:00.587561 17915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "2280e05dcefd4289864773b8a130ed1d"
format_stamp: "Formatted at 2026-09-25 01:35:00 on dist-test-slave-h6hf"
server_key: "b9cb0ef837e72cc1647ccf913efba125"
server_key_iv: "99cbf3ce32ec87efdc2c50729451381a"
server_key_version: "encryptionkey@0"
I20260925 01:35:00.592952 17915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:35:00.596495 17930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:00.597922 17915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:35:00.598114 17915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "2280e05dcefd4289864773b8a130ed1d"
format_stamp: "Formatted at 2026-09-25 01:35:00 on dist-test-slave-h6hf"
server_key: "b9cb0ef837e72cc1647ccf913efba125"
server_key_iv: "99cbf3ce32ec87efdc2c50729451381a"
server_key_version: "encryptionkey@0"
I20260925 01:35:00.598330 17915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:00.617880 17915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:00.622424 17915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:00.622726 17915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:00.642336 17915 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:43913
I20260925 01:35:00.642369 17982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:43913 every 8 connection(s)
I20260925 01:35:00.643813 17915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:35:00.648079 17983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:00.649978 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 17915
I20260925 01:35:00.650183 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:35:00.650607 28288 external_mini_cluster.cc:1428] Setting key 93e124d21dcd06eb4e56e5bb14d18b0f
I20260925 01:35:00.661417 17983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d: Bootstrap starting.
Sep 25 01:35:00 dist-test-slave-h6hf krb5kdc[17902](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:35:00.664739 17983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:00.665884 17983 log.cc:824] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:35:00.669467 17983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d: No bootstrap required, opened a new log
I20260925 01:35:00.674420 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:00.652577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51547 (local address 127.27.160.62:43913)
0925 01:35:00.653464 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:00.653476 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:00.653517 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:35:00.655169 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:00.655195 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:00.655645 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:00.656054 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:00.657599 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:00.658947 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:00.660330 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:00.661020 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:00.661238 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:00.665590 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:00.665602 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:00.665624 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:00.665667 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:00.669519 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:00.670278 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:00.670288 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:00.670296 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:00.670498 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:00.670894 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:00.670898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:00.670901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:00.671709 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:00.672017 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:00.672416 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 01:35:00.672929 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":364,"threads_started":1}
I20260925 01:35:00.678423 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:35:00.679575 17983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2280e05dcefd4289864773b8a130ed1d" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43913 } }
I20260925 01:35:00.679972 17983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:35:00.680087 17983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2280e05dcefd4289864773b8a130ed1d, State: Initialized, Role: FOLLOWER
I20260925 01:35:00.680653 17983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2280e05dcefd4289864773b8a130ed1d" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43913 } }
I20260925 01:35:00.680929 17983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:35:00.681108 17983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:35:00.681352 17983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:00.684139 17983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2280e05dcefd4289864773b8a130ed1d" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43913 } }
I20260925 01:35:00.684643 17983 leader_election.cc:302] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2280e05dcefd4289864773b8a130ed1d; no voters:
I20260925 01:35:00.685374 17983 leader_election.cc:288] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:35:00.685472 17993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:35:00.686126 17993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [term 1 LEADER]: Becoming Leader. State: Replica: 2280e05dcefd4289864773b8a130ed1d, State: Running, Role: LEADER
I20260925 01:35:00.686935 17993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2280e05dcefd4289864773b8a130ed1d" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43913 } }
I20260925 01:35:00.690145 17993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2280e05dcefd4289864773b8a130ed1d. Latest consensus state: current_term: 1 leader_uuid: "2280e05dcefd4289864773b8a130ed1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2280e05dcefd4289864773b8a130ed1d" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43913 } } }
I20260925 01:35:00.690640 17993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:00.690892 17995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2280e05dcefd4289864773b8a130ed1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2280e05dcefd4289864773b8a130ed1d" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43913 } } }
I20260925 01:35:00.693738 17995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:00.697686 17983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:35:00.703998 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:00.683132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47225 (local address 127.27.160.62:43913)
0925 01:35:00.683305 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:00.683309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:00.683762 (+ 453us) server_negotiation.cc:407] Connection header received
0925 01:35:00.683958 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:00.683963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:00.684041 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:00.684157 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:00.685855 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:00.686949 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:00.688415 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:00.689069 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:00.689278 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:00.693011 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:00.693022 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:00.693030 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:00.693082 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:00.698496 (+ 5414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:00.699735 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:00.699743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:00.699749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:00.699932 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:00.700498 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:00.700505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:00.700511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:00.701007 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:00.702960 (+ 1953us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:00.703145 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 01:35:00.703451 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:35:00.706806 17997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:35:00.711503 17997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:35:00.720816 17997 catalog_manager.cc:1347] Generated new cluster ID: 4631a5e185264d798b77e8085d7f9f0e
I20260925 01:35:00.720973 17997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:35:00.744372 17997 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:35:00.745888 17997 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:35:00.754323 17997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2280e05dcefd4289864773b8a130ed1d: Generated new TSK 0
I20260925 01:35:00.755200 17997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:35:00.764245 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 17915
I20260925 01:35:00.765362 17978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:35:00.858839 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 17915
2026-09-25T01:35:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2125 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:35:00 dist-test-slave-h6hf krb5kdc[18015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:35:00 dist-test-slave-h6hf krb5kdc[18015](info): set up 2 sockets
Sep 25 01:35:00 dist-test-slave-h6hf krb5kdc[18015](info): commencing operation
krb5kdc: starting...
W20260925 01:35:02.030783 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:35:02 dist-test-slave-h6hf krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:35:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:35:02.314626 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:42307
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:35:02.663499 18028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:02.663815 18028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:02.664021 18028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:02.671548 18028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:02.671689 18028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:02.671772 18028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:02.671868 18028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:02.671943 18028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:02.682214 18028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:42307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:02.683634 18028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:02.684985 18028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:35:02.697716 18033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:02.697691 18034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:02.697757 18036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:02.698673 18028 server_base.cc:1034] running on GCE node
I20260925 01:35:02.699496 18028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:02.701059 18028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:02.702407 18028 hybrid_clock.cc:648] HybridClock initialized: now 1790300102702360 us; error 33 us; skew 500 ppm
I20260925 01:35:02.702905 18028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:02 dist-test-slave-h6hf krb5kdc[18015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300102, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:02.706380 18028 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:35:02.707983 18028 webserver.cc:460] Webserver started at http://127.27.160.62:46611/ using document root <none> and password file <none>
I20260925 01:35:02.708706 18028 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:02.708812 18028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:02.709075 18028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:02.712369 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "a1cbb8d9840b4a0aba94703a70e9346e"
format_stamp: "Formatted at 2026-09-25 01:35:02 on dist-test-slave-h6hf"
server_key: "20a575a6ba382e9a67615e3220332445"
server_key_iv: "fa5fa36f576c3989988fa23f90af9490"
server_key_version: "encryptionkey@0"
I20260925 01:35:02.713109 18028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "a1cbb8d9840b4a0aba94703a70e9346e"
format_stamp: "Formatted at 2026-09-25 01:35:02 on dist-test-slave-h6hf"
server_key: "20a575a6ba382e9a67615e3220332445"
server_key_iv: "fa5fa36f576c3989988fa23f90af9490"
server_key_version: "encryptionkey@0"
I20260925 01:35:02.718693 18028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:35:02.722400 18043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:02.723915 18028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:35:02.724066 18028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "a1cbb8d9840b4a0aba94703a70e9346e"
format_stamp: "Formatted at 2026-09-25 01:35:02 on dist-test-slave-h6hf"
server_key: "20a575a6ba382e9a67615e3220332445"
server_key_iv: "fa5fa36f576c3989988fa23f90af9490"
server_key_version: "encryptionkey@0"
I20260925 01:35:02.724242 18028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:02.748719 18028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:02.754019 18028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:02.754343 18028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:02.774421 18028 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:42307
I20260925 01:35:02.774428 18095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:42307 every 8 connection(s)
I20260925 01:35:02.775935 18028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:35:02.779731 18096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:02.782531 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 18028
I20260925 01:35:02.782727 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:35:02.783164 28288 external_mini_cluster.cc:1428] Setting key 0a8f5f8c901204b04d4b74180a190e6f
I20260925 01:35:02.792313 18096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e: Bootstrap starting.
I20260925 01:35:02.796149 18096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:02.797295 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:02.784968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54889 (local address 127.27.160.62:42307)
0925 01:35:02.785875 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:02.785890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:02.785932 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:35:02.787687 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:02.787712 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:02.788135 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:02.788493 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:02.789106 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:02.790574 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:02.792495 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:02.793331 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:02.793471 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:02.794855 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:02.794863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:02.794877 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:35:02.795426 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:02.795503 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:35:02.795528 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:02.795607 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:02.795892 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 01:35:02.796171 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":443,"threads_started":1}
I20260925 01:35:02.797298 18096 log.cc:824] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:35:02.800724 18096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e: No bootstrap required, opened a new log
I20260925 01:35:02.801645 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:35:02.811069 18096 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbb8d9840b4a0aba94703a70e9346e" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42307 } }
I20260925 01:35:02.811584 18096 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:35:02.811718 18096 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cbb8d9840b4a0aba94703a70e9346e, State: Initialized, Role: FOLLOWER
I20260925 01:35:02.812310 18096 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbb8d9840b4a0aba94703a70e9346e" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42307 } }
I20260925 01:35:02.812551 18096 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:35:02.812727 18096 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:35:02.812932 18096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:02.813807 18099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:02.806982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38251 (local address 127.27.160.62:42307)
0925 01:35:02.807179 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:02.807186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:02.807675 (+ 489us) server_negotiation.cc:407] Connection header received
0925 01:35:02.807828 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:02.807833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:02.807908 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:02.808027 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:02.808541 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:02.809728 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:02.810984 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:02.811635 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:02.811780 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:02.813014 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:02.813022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:02.813027 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:35:02.813052 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:02.813105 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:35:02.813161 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:02.813276 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:02.813371 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 01:35:02.813464 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:35:02.815999 18096 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbb8d9840b4a0aba94703a70e9346e" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42307 } }
I20260925 01:35:02.816551 18096 leader_election.cc:302] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cbb8d9840b4a0aba94703a70e9346e; no voters:
I20260925 01:35:02.817502 18096 leader_election.cc:288] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:35:02.817665 18106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:35:02.818733 18106 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [term 1 LEADER]: Becoming Leader. State: Replica: a1cbb8d9840b4a0aba94703a70e9346e, State: Running, Role: LEADER
I20260925 01:35:02.819360 18106 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbb8d9840b4a0aba94703a70e9346e" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42307 } }
I20260925 01:35:02.820163 18096 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:35:02.822585 18107 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1cbb8d9840b4a0aba94703a70e9346e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbb8d9840b4a0aba94703a70e9346e" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42307 } } }
I20260925 01:35:02.823145 18107 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:02.823328 18108 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cbb8d9840b4a0aba94703a70e9346e. Latest consensus state: current_term: 1 leader_uuid: "a1cbb8d9840b4a0aba94703a70e9346e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbb8d9840b4a0aba94703a70e9346e" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 42307 } } }
I20260925 01:35:02.823639 18108 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:02.824796 18111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:35:02.831583 18111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:35:02.840587 18111 catalog_manager.cc:1347] Generated new cluster ID: 65c63dc49f984a3781f8925caae90507
I20260925 01:35:02.840725 18111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:35:02.853431 18111 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:35:02.854821 18111 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:35:02.864298 18111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1cbb8d9840b4a0aba94703a70e9346e: Generated new TSK 0
I20260925 01:35:02.865275 18111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:35:02.876910 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18028
W20260925 01:35:02.878155 18091 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54889
W20260925 01:35:02.878927 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18028: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:35:02.879088 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 18028
2026-09-25T01:35:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:35:02 dist-test-slave-h6hf krb5kdc[18127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:35:02 dist-test-slave-h6hf krb5kdc[18127](info): set up 2 sockets
Sep 25 01:35:02 dist-test-slave-h6hf krb5kdc[18127](info): commencing operation
krb5kdc: starting...
W20260925 01:35:04.026721 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:35:04 dist-test-slave-h6hf krb5kdc[18127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:35:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:35:04.324095 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:35519
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:35:04.668452 18140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:04.668777 18140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:04.668871 18140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:04.676427 18140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:04.676564 18140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:04.676612 18140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:04.676672 18140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:04.676719 18140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:04.686863 18140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:35519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:04.688117 18140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:04.689443 18140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:35:04.702062 18145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:04.702092 18146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:04.702878 18148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:04.703382 18140 server_base.cc:1034] running on GCE node
I20260925 01:35:04.704234 18140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:04.705858 18140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:04.707100 18140 hybrid_clock.cc:648] HybridClock initialized: now 1790300104707037 us; error 58 us; skew 500 ppm
I20260925 01:35:04.707485 18140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:04 dist-test-slave-h6hf krb5kdc[18127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300104, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:04.710961 18140 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:35:04.712560 18140 webserver.cc:460] Webserver started at http://127.27.160.62:35633/ using document root <none> and password file <none>
I20260925 01:35:04.713380 18140 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:04.713505 18140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:04.713783 18140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:04.716893 18140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "6d32bf7bd1744a37b596b5241cb49e6a"
format_stamp: "Formatted at 2026-09-25 01:35:04 on dist-test-slave-h6hf"
server_key: "1f94e6f3ff27417a88943eee167842d3"
server_key_iv: "fbe4d86a98c04ff1c3c5d8a6cb109261"
server_key_version: "encryptionkey@0"
I20260925 01:35:04.717752 18140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "6d32bf7bd1744a37b596b5241cb49e6a"
format_stamp: "Formatted at 2026-09-25 01:35:04 on dist-test-slave-h6hf"
server_key: "1f94e6f3ff27417a88943eee167842d3"
server_key_iv: "fbe4d86a98c04ff1c3c5d8a6cb109261"
server_key_version: "encryptionkey@0"
I20260925 01:35:04.723055 18140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 01:35:04.726547 18155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:04.728052 18140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:35:04.728216 18140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "6d32bf7bd1744a37b596b5241cb49e6a"
format_stamp: "Formatted at 2026-09-25 01:35:04 on dist-test-slave-h6hf"
server_key: "1f94e6f3ff27417a88943eee167842d3"
server_key_iv: "fbe4d86a98c04ff1c3c5d8a6cb109261"
server_key_version: "encryptionkey@0"
I20260925 01:35:04.728408 18140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:04.734913 18140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:04.739547 18140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:04.739820 18140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:04.758576 18140 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:35519
I20260925 01:35:04.758610 18206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:35519 every 8 connection(s)
I20260925 01:35:04.759976 18140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:35:04.761168 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 18140
I20260925 01:35:04.761376 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:35:04.761961 28288 external_mini_cluster.cc:1428] Setting key 35beccd9d50d6b50a2be14c43c5268f9
I20260925 01:35:04.764588 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:04.769927 18211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:04.763819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43381 (local address 127.27.160.62:35519)
0925 01:35:04.764622 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:04.764657 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:04.764708 (+ 51us) server_negotiation.cc:407] Connection header received
0925 01:35:04.766402 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:04.766424 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:04.766831 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:04.767179 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:04.768216 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:04.768222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:04.768234 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:35:04.768248 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:04.768330 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:35:04.768351 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:04.768518 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:04.768695 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 01:35:04.768942 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":335,"threads_started":1}
I20260925 01:35:04.773581 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:35:04.776928 18207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a: Bootstrap starting.
I20260925 01:35:04.781178 18207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:04.781517 18211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:04.778876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57681 (local address 127.27.160.62:35519)
0925 01:35:04.779027 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:04.779032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:04.779668 (+ 636us) server_negotiation.cc:407] Connection header received
0925 01:35:04.779796 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:04.779799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:04.779845 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:04.779920 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:04.780831 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:04.780835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:04.780838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:35:04.780853 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:04.780890 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:35:04.780904 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:04.780987 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:04.781198 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 01:35:04.781245 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 01:35:04.782414 18207 log.cc:824] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:35:04.785050 18207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a: No bootstrap required, opened a new log
I20260925 01:35:04.792799 18207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d32bf7bd1744a37b596b5241cb49e6a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35519 } }
I20260925 01:35:04.793221 18207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:35:04.793366 18207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d32bf7bd1744a37b596b5241cb49e6a, State: Initialized, Role: FOLLOWER
I20260925 01:35:04.793936 18207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d32bf7bd1744a37b596b5241cb49e6a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35519 } }
I20260925 01:35:04.794162 18207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:35:04.794318 18207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:35:04.794489 18207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:04.796716 18207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d32bf7bd1744a37b596b5241cb49e6a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35519 } }
I20260925 01:35:04.797299 18207 leader_election.cc:302] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d32bf7bd1744a37b596b5241cb49e6a; no voters:
I20260925 01:35:04.797958 18207 leader_election.cc:288] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:35:04.798084 18218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:35:04.799101 18218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [term 1 LEADER]: Becoming Leader. State: Replica: 6d32bf7bd1744a37b596b5241cb49e6a, State: Running, Role: LEADER
I20260925 01:35:04.799693 18218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d32bf7bd1744a37b596b5241cb49e6a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35519 } }
I20260925 01:35:04.800580 18207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:35:04.802542 18220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d32bf7bd1744a37b596b5241cb49e6a. Latest consensus state: current_term: 1 leader_uuid: "6d32bf7bd1744a37b596b5241cb49e6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d32bf7bd1744a37b596b5241cb49e6a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35519 } } }
I20260925 01:35:04.802956 18220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:04.803728 18219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d32bf7bd1744a37b596b5241cb49e6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d32bf7bd1744a37b596b5241cb49e6a" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 35519 } } }
I20260925 01:35:04.804123 18219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:04.805848 18224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:35:04.811586 18224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:35:04.821523 18224 catalog_manager.cc:1347] Generated new cluster ID: 409aa162345443d9b9004732fd456685
I20260925 01:35:04.821645 18224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:35:04.858798 18224 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:35:04.859341 18224 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:35:04.867511 18224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d32bf7bd1744a37b596b5241cb49e6a: Generated new TSK 0
I20260925 01:35:04.868607 18224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:35:04.875979 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18140
W20260925 01:35:04.877094 18202 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43381
W20260925 01:35:04.877887 28288 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18140: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:35:04.878046 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 18140
2026-09-25T01:35:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1999 ms)
[----------] 18 tests from AuthTokenIssuingTest (36772 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:35:04 dist-test-slave-h6hf krb5kdc[18239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:35:04 dist-test-slave-h6hf krb5kdc[18239](info): set up 2 sockets
Sep 25 01:35:04 dist-test-slave-h6hf krb5kdc[18239](info): commencing operation
krb5kdc: starting...
W20260925 01:35:06.010825 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:35:06 dist-test-slave-h6hf krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:35:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:35:06.288861 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:43437
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:43437,127.27.160.61:33331,127.27.160.60:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:35:06.634898 18252 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:35:06.635298 18252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:06.635442 18252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:06.635597 18252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:06.643116 18252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:06.643224 18252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:06.643327 18252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:06.643409 18252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:06.643502 18252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:06.653811 18252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:43437,127.27.160.61:33331,127.27.160.60:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:43437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:06.655311 18252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:06.656697 18252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:35:06.668941 18257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:06.669070 18258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:06.669098 18260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:06.669962 18252 server_base.cc:1034] running on GCE node
I20260925 01:35:06.670811 18252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:06.672348 18252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:06.673578 18252 hybrid_clock.cc:648] HybridClock initialized: now 1790300106673471 us; error 86 us; skew 500 ppm
I20260925 01:35:06.673977 18252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:06 dist-test-slave-h6hf krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300106, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:06.680976 18252 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:35:06.682555 18252 webserver.cc:460] Webserver started at http://127.27.160.62:35195/ using document root <none> and password file <none>
I20260925 01:35:06.683244 18252 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:06.683334 18252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:06.683617 18252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:06.686776 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "5b61b9c36c4b4c9cac6ecb6591736f84"
format_stamp: "Formatted at 2026-09-25 01:35:06 on dist-test-slave-h6hf"
server_key: "67813c9f8a280342a1841b8ceac463c5"
server_key_iv: "2912526452bf2ce000ffd614f99c8773"
server_key_version: "encryptionkey@0"
I20260925 01:35:06.687533 18252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "5b61b9c36c4b4c9cac6ecb6591736f84"
format_stamp: "Formatted at 2026-09-25 01:35:06 on dist-test-slave-h6hf"
server_key: "67813c9f8a280342a1841b8ceac463c5"
server_key_iv: "2912526452bf2ce000ffd614f99c8773"
server_key_version: "encryptionkey@0"
I20260925 01:35:06.692659 18252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:35:06.696429 18267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:06.698082 18252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:35:06.698294 18252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "5b61b9c36c4b4c9cac6ecb6591736f84"
format_stamp: "Formatted at 2026-09-25 01:35:06 on dist-test-slave-h6hf"
server_key: "67813c9f8a280342a1841b8ceac463c5"
server_key_iv: "2912526452bf2ce000ffd614f99c8773"
server_key_version: "encryptionkey@0"
I20260925 01:35:06.698540 18252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:06.717692 18252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:06.722707 18252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:06.723003 18252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:06.741366 18252 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:43437
I20260925 01:35:06.741387 18318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:43437 every 8 connection(s)
I20260925 01:35:06.742885 18252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:35:06.746171 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 18252
I20260925 01:35:06.746407 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:35:06.747251 28288 external_mini_cluster.cc:1428] Setting key 4dab16b5a00229688bae31a6c0ee49ef
I20260925 01:35:06.747551 18319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:06.757985 18319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:06 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300106, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.61@KRBTEST.COM" created.
I20260925 01:35:06.782568 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:06.761856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43437 (local address 127.0.0.1:37908)
0925 01:35:06.762744 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:06.762783 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:06.764960 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:06.765893 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:06.765900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:06.766608 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:06.766939 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:06.766955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:06.768496 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:06.768501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:06.769648 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:06.769659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:06.770560 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:06.770567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:06.770840 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:06.771591 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:06.771633 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:06.774461 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:06.778055 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:06.778068 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:06.778082 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:06.778574 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:06.779099 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:06.779103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:06.779105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:06.779281 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:06.780433 (+ 1152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:06.780439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:06.780642 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 01:35:06.780854 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 01:35:06.781401 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":4,"thread_start_us":401,"threads_started":1}
I20260925 01:35:06.782621 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:06.763367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37908 (local address 127.27.160.62:43437)
0925 01:35:06.764153 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:06.764162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:06.764221 (+ 59us) server_negotiation.cc:407] Connection header received
0925 01:35:06.765419 (+ 1198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:06.765424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:06.765665 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:06.765873 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:06.767155 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:06.768335 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:06.769782 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:06.770331 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:06.770528 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:06.774682 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:06.774696 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:06.774708 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:06.774736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:06.777908 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:06.778726 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:06.778737 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:06.778743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:06.778955 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:06.779421 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:06.779429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:06.779436 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:06.780270 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:06.780544 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:06.781085 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 01:35:06.781613 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":225,"server-negotiator.queue_time_us":583,"thread_start_us":418,"threads_started":1}
I20260925 01:35:06.786101 18319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:35:06.787299 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:06.786887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52084)
0925 01:35:06.787062 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:06.787188 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 01:35:06.787657 18271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
I20260925 01:35:06.789362 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:06.788934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52100)
0925 01:35:06.789108 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:06.789239 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:35:06.789587 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:33331: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab.
Entry for principal kudu/127.27.160.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab.
I20260925 01:35:06.813426 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.61:33331
--webserver_interface=127.27.160.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab
--principal=kudu/127.27.160.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:43437,127.27.160.61:33331,127.27.160.60:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:35:06.838940 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } attempt: 1
I20260925 01:35:06.840260 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:06.839771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52106)
0925 01:35:06.839985 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:06.840110 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 01:35:06.841392 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:06.840960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52110)
0925 01:35:06.841176 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:06.841261 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:35:06.841575 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:33331: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
I20260925 01:35:06.909829 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } attempt: 2
I20260925 01:35:06.911063 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:06.910563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52122)
0925 01:35:06.910833 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:06.910922 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 01:35:06.912158 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:06.911777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52134)
0925 01:35:06.911932 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:06.912013 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:35:06.912353 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:33331: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
I20260925 01:35:07.003609 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } attempt: 3
I20260925 01:35:07.004776 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.004294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52148)
0925 01:35:07.004514 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.004613 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 01:35:07.005875 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.005499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52150)
0925 01:35:07.005652 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.005734 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:35:07.006080 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:33331: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
I20260925 01:35:07.149296 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } attempt: 4
I20260925 01:35:07.150489 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.149920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52154)
0925 01:35:07.150211 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.150320 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260925 01:35:07.151440 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.151100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52158)
0925 01:35:07.151245 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.151331 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 01:35:07.151616 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:33331: Network error: Client connection negotiation failed: client connection to 127.27.160.61:33331: connect: Connection refused (error 111)
W20260925 01:35:07.156639 18326 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:35:07.157009 18326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:07.157176 18326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:07.157332 18326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:07.164685 18326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:07.164816 18326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:07.164921 18326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:07.165021 18326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:07.165110 18326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:07.175506 18326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:43437,127.27.160.61:33331,127.27.160.60:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab
--principal=kudu/127.27.160.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.61:33331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.160.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:07.176893 18326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:07.178323 18326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:35:07.190488 18332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:07.190553 18331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:07.190529 18334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:07.191406 18326 server_base.cc:1034] running on GCE node
I20260925 01:35:07.192229 18326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:07.193861 18326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:07.195120 18326 hybrid_clock.cc:648] HybridClock initialized: now 1790300107195044 us; error 48 us; skew 500 ppm
I20260925 01:35:07.195473 18326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:07.199167 18326 init.cc:377] Logged in from keytab as kudu/127.27.160.61@KRBTEST.COM (short username kudu)
I20260925 01:35:07.200699 18326 webserver.cc:460] Webserver started at http://127.27.160.61:33801/ using document root <none> and password file <none>
I20260925 01:35:07.201500 18326 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:07.201639 18326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:07.201963 18326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:07.205039 18326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/data/instance:
uuid: "c1923c26647e4e56808bd86bf1c914b3"
format_stamp: "Formatted at 2026-09-25 01:35:07 on dist-test-slave-h6hf"
server_key: "42148c6aa2b3520b8f2f2c7ed0736725"
server_key_iv: "2e6ac7cd716a1f0480e40bf8893808fe"
server_key_version: "encryptionkey@0"
I20260925 01:35:07.205916 18326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/wal/instance:
uuid: "c1923c26647e4e56808bd86bf1c914b3"
format_stamp: "Formatted at 2026-09-25 01:35:07 on dist-test-slave-h6hf"
server_key: "42148c6aa2b3520b8f2f2c7ed0736725"
server_key_iv: "2e6ac7cd716a1f0480e40bf8893808fe"
server_key_version: "encryptionkey@0"
I20260925 01:35:07.211082 18326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:35:07.214599 18341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:07.215955 18326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:35:07.216161 18326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/wal
uuid: "c1923c26647e4e56808bd86bf1c914b3"
format_stamp: "Formatted at 2026-09-25 01:35:07 on dist-test-slave-h6hf"
server_key: "42148c6aa2b3520b8f2f2c7ed0736725"
server_key_iv: "2e6ac7cd716a1f0480e40bf8893808fe"
server_key_version: "encryptionkey@0"
I20260925 01:35:07.216382 18326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:07.228968 18326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:07.233506 18326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:07.233798 18326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:07.252971 18326 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.61:33331
I20260925 01:35:07.252995 18392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.61:33331 every 8 connection(s)
I20260925 01:35:07.254500 18326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
I20260925 01:35:07.258486 18393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:07.262553 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 18326
I20260925 01:35:07.262760 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-1/wal/instance
I20260925 01:35:07.263214 28288 external_mini_cluster.cc:1428] Setting key 683ea64088997821a5050654fa594d0f
I20260925 01:35:07.265524 18393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.60@KRBTEST.COM; defaulting to no policy
I20260925 01:35:07.286965 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.268441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.27.160.62:43437)
0925 01:35:07.268669 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.268676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.270016 (+ 1340us) server_negotiation.cc:407] Connection header received
0925 01:35:07.272074 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.272080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.272179 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.272296 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.274005 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.274729 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.276313 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.276750 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.276889 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.280726 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.280731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.280735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.280770 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.283636 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.284445 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.284450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.284452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.284581 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.285029 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.285041 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.285044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.285387 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.285626 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.286389 (+ 763us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.286633 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:35:07.288064 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.268879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43437 (local address 127.0.0.1:37924)
0925 01:35:07.269795 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.269852 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:07.271747 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:07.272574 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:07.272584 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:07.273198 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:07.273811 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.273830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.274918 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.274922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.276144 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.276157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.276997 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.277005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.277340 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.278083 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:07.278107 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:07.280359 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:07.283794 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.283804 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.283823 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.284301 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.284697 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.284707 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.284710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.284906 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.285606 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:07.285615 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:07.285974 (+ 359us) client_negotiation.cc:769] Sending connection context
0925 01:35:07.286283 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 01:35:07.286824 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":430,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.60@KRBTEST.COM" created.
I20260925 01:35:07.290216 18393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:35:07.305950 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.291145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52172)
0925 01:35:07.291322 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.291349 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:07.291442 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:07.292689 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:07.292694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:07.293252 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:07.293391 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.293400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.294840 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.294848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.295709 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.295720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.296557 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.296563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.296742 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.297469 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:07.297491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:07.299813 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:07.303335 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.303342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.303345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.303726 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.304121 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.304125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.304129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.304257 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.304842 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:07.304852 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:07.305000 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 01:35:07.305209 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 01:35:07.305520 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 01:35:07.306020 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.291416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.27.160.61:33331)
0925 01:35:07.292178 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.292184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.292209 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:35:07.292256 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.292262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.292439 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.292710 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.293583 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.294712 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.295854 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.296402 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.296540 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.300062 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.300074 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.300090 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.300118 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.303204 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.303862 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.303874 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.303876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.303983 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.304371 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.304375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.304379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.304720 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.304928 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.305370 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.305622 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":608,"thread_start_us":452,"threads_started":1}
I20260925 01:35:07.308815 18393 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:35:07.309990 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.309496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40854)
0925 01:35:07.309728 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.309839 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260925 01:35:07.310241 18344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
I20260925 01:35:07.311856 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.311479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40858)
0925 01:35:07.311654 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.311743 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:35:07.312062 18393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab.
Entry for principal kudu/127.27.160.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab.
I20260925 01:35:07.324878 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.60:33861
--webserver_interface=127.27.160.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab
--principal=kudu/127.27.160.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:43437,127.27.160.61:33331,127.27.160.60:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:35:07.361368 18393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 1
I20260925 01:35:07.362645 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.362176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40874)
0925 01:35:07.362380 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.362490 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 01:35:07.363718 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.363359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40878)
0925 01:35:07.363510 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.363587 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 01:35:07.363973 18393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
I20260925 01:35:07.432247 18393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 2
I20260925 01:35:07.433653 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.433061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40886)
0925 01:35:07.433374 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.433475 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260925 01:35:07.434823 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.434458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40892)
0925 01:35:07.434619 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.434695 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 01:35:07.435115 18393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
I20260925 01:35:07.450881 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } attempt: 5
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300106, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:35:07.464552 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.451674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52174)
0925 01:35:07.451895 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.451919 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:07.452071 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:07.452445 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:07.452452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:07.452869 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:07.453057 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.453067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.454152 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.454157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.455055 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.455067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.455839 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.455846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.456080 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.456762 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:07.456780 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:07.459017 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:07.462151 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.462157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.462161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.462562 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.462938 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.462943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.462947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.463087 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.463631 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:07.463636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:07.463843 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 01:35:07.464024 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 01:35:07.464291 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 01:35:07.464875 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.451808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.27.160.61:33331)
0925 01:35:07.452055 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.452061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.452083 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:35:07.452218 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.452223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.452289 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.452401 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.453257 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.453986 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.455224 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.455684 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.455823 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.459240 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.459245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.459249 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.459282 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.462000 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.462687 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.462693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.462697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.462824 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.463206 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.463211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.463214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.463500 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.463653 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.464223 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.464485 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 01:35:07.465929 18319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:35:07.467325 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.466719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40896)
0925 01:35:07.467011 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.467115 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260925 01:35:07.468454 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.468144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40908)
0925 01:35:07.468279 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.468360 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 01:35:07.468751 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
I20260925 01:35:07.519973 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 1
I20260925 01:35:07.521265 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.520731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40922)
0925 01:35:07.520920 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.521014 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 01:35:07.522274 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.521934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40934)
0925 01:35:07.522079 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.522160 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 01:35:07.522490 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
I20260925 01:35:07.526341 18393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 3
I20260925 01:35:07.527415 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.526977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40946)
0925 01:35:07.527172 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.527267 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 01:35:07.528457 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.528043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40952)
0925 01:35:07.528288 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.528358 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260925 01:35:07.528642 18393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
I20260925 01:35:07.590740 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 2
I20260925 01:35:07.592072 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.591480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40956)
0925 01:35:07.591793 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.591910 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260925 01:35:07.593102 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.592737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40970)
0925 01:35:07.592912 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.592991 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 01:35:07.593380 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
I20260925 01:35:07.671857 18393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 4
I20260925 01:35:07.673094 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.672593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:40986)
0925 01:35:07.672851 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.672939 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 01:35:07.674250 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.673927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:41002)
0925 01:35:07.674073 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.674150 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 01:35:07.674437 18393 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
W20260925 01:35:07.686923 18400 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:35:07.687311 18400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:07.687438 18400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:07.687572 18400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:07.695132 18400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:07.695266 18400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:07.695402 18400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:07.695492 18400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:07.695605 18400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:07.699584 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 3
I20260925 01:35:07.700691 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.700207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:41004)
0925 01:35:07.700444 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.700533 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 01:35:07.701737 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.701327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:41014)
0925 01:35:07.701555 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.701640 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260925 01:35:07.701929 18319 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:33861: Network error: Client connection negotiation failed: client connection to 127.27.160.60:33861: connect: Connection refused (error 111)
I20260925 01:35:07.706422 18400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:43437,127.27.160.61:33331,127.27.160.60:33861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab
--principal=kudu/127.27.160.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.60:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.160.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:07.707836 18400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:07.709317 18400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:35:07.721884 18405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:07.721921 18408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:07.721843 18406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:07.723136 18400 server_base.cc:1034] running on GCE node
I20260925 01:35:07.723894 18400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:07.725355 18400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:07.726581 18400 hybrid_clock.cc:648] HybridClock initialized: now 1790300107726494 us; error 62 us; skew 500 ppm
I20260925 01:35:07.726914 18400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:07.730011 18400 init.cc:377] Logged in from keytab as kudu/127.27.160.60@KRBTEST.COM (short username kudu)
I20260925 01:35:07.731536 18400 webserver.cc:460] Webserver started at http://127.27.160.60:45517/ using document root <none> and password file <none>
I20260925 01:35:07.732255 18400 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:07.732390 18400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:07.732762 18400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:07.735994 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/data/instance:
uuid: "0e9ce18183314ee9bbb5517a6e74eeb6"
format_stamp: "Formatted at 2026-09-25 01:35:07 on dist-test-slave-h6hf"
server_key: "fd497aaafeb6a98183b926599d6b3bd7"
server_key_iv: "08c437a272c302c98e431e534b1541e2"
server_key_version: "encryptionkey@0"
I20260925 01:35:07.736840 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/wal/instance:
uuid: "0e9ce18183314ee9bbb5517a6e74eeb6"
format_stamp: "Formatted at 2026-09-25 01:35:07 on dist-test-slave-h6hf"
server_key: "fd497aaafeb6a98183b926599d6b3bd7"
server_key_iv: "08c437a272c302c98e431e534b1541e2"
server_key_version: "encryptionkey@0"
I20260925 01:35:07.742269 18400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:35:07.745869 18415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:07.747315 18400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 01:35:07.747505 18400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/wal
uuid: "0e9ce18183314ee9bbb5517a6e74eeb6"
format_stamp: "Formatted at 2026-09-25 01:35:07 on dist-test-slave-h6hf"
server_key: "fd497aaafeb6a98183b926599d6b3bd7"
server_key_iv: "08c437a272c302c98e431e534b1541e2"
server_key_version: "encryptionkey@0"
I20260925 01:35:07.747784 18400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:07.764206 18400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:07.768216 18400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:07.768500 18400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:07.786980 18400 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.60:33861
I20260925 01:35:07.786989 18467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.60:33861 every 8 connection(s)
I20260925 01:35:07.788727 18400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
I20260925 01:35:07.792549 18468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:07.794240 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 18400
I20260925 01:35:07.794443 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790299737515179-28288-0/minicluster-data/master-2/wal/instance
I20260925 01:35:07.794867 28288 external_mini_cluster.cc:1428] Setting key d7635080d49c83aba9930c73b74111fd
I20260925 01:35:07.801934 18468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:35:07.819592 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.796766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.27.160.60:33861)
0925 01:35:07.797620 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.797631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.797677 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:35:07.799324 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.799350 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.799763 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.800081 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.802051 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.803644 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.805353 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.806150 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.806344 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.810955 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.810963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.810974 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.811041 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.815021 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.815928 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.815952 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.815959 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.816136 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.816571 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.816580 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.816582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.817245 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.817554 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.818027 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.818483 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":407,"threads_started":1}
I20260925 01:35:07.821352 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.804812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.27.160.62:43437)
0925 01:35:07.805370 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.805376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.805999 (+ 623us) server_negotiation.cc:407] Connection header received
0925 01:35:07.806180 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.806187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.806285 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.806392 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.807899 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.808673 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.810306 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.810945 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.811040 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.815316 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.815324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.815332 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.815376 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.818207 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.819061 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.819073 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.819076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.819207 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.819643 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.819652 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.819654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.820034 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.820203 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.820706 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.820923 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":345,"threads_started":1}
I20260925 01:35:07.821393 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.804808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:43437 (local address 127.0.0.1:37926)
0925 01:35:07.805845 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.805893 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:07.806049 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:07.806476 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:07.806484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:07.807374 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:07.807730 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.807746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.808836 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.808841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.810153 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.810166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.811139 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.811146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.811364 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.812100 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:07.812124 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:07.815022 (+ 2898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:07.818495 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.818505 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.818510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.818930 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.819320 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.819325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.819329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.819508 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.820164 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:07.820172 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:07.820370 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 01:35:07.820538 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 01:35:07.820890 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":619,"threads_started":1}
I20260925 01:35:07.823738 18468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:35:07.838604 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.824461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:33331 (local address 127.0.0.1:52182)
0925 01:35:07.824657 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.824671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:07.824747 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:07.825228 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:07.825233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:07.825635 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:07.825833 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.825846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.826985 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.826989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.828029 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.828043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.828793 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.828797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.828921 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.829721 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:07.829739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:07.831957 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:07.835669 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.835675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.835680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.836196 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.836711 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.836714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.836717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.836853 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.837524 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:07.837527 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:07.837788 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 01:35:07.837946 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 01:35:07.838266 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:35:07.838686 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.824549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.27.160.61:33331)
0925 01:35:07.824815 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.824823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.824848 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:35:07.824911 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.824918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.825002 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.825163 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.825992 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.826835 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.828176 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.828642 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.828777 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.832248 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.832255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.832260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.832303 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.835508 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.836365 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.836371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.836374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.836550 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.836971 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.836978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.836984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.837401 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.837559 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.838025 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.838285 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260925 01:35:07.839221 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.823095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.27.160.62:43437)
0925 01:35:07.823261 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.823265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.823511 (+ 246us) server_negotiation.cc:407] Connection header received
0925 01:35:07.823723 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.823731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.823825 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.823928 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.825600 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.826450 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.827867 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.828437 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.828542 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.833251 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.833257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.833260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.833294 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.836117 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.837023 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.837026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.837029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.837228 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.837626 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.837631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.837634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.837993 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.838296 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.838640 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.838895 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:35:07.840000 18468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:35:07.854522 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.840921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.27.160.60:33861)
0925 01:35:07.841162 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.841169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.841366 (+ 197us) server_negotiation.cc:407] Connection header received
0925 01:35:07.841488 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.841493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.841558 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.841715 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.842669 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.843549 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.845009 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.845504 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.845656 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.849004 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.849008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.849012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.849039 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.851821 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.852657 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.852661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.852663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.852780 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.853248 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.853252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.853254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.853525 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.853638 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.854090 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.854272 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260925 01:35:07.854794 18473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.840920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:41030)
0925 01:35:07.841281 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.841299 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:07.841392 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:07.841771 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:07.841781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:07.842273 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:07.842481 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.842495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.843675 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.843678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.844867 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.844883 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.845699 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.845703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.845855 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.846459 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:07.846476 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:07.848824 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:07.851968 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.851973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.851977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.852540 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.852893 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.852898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.852902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.853069 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.853655 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:07.853660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:07.853831 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 01:35:07.854011 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 01:35:07.854425 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":127}
I20260925 01:35:07.857478 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.840970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.27.160.61:33331)
0925 01:35:07.841144 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.841150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.841313 (+ 163us) server_negotiation.cc:407] Connection header received
0925 01:35:07.841481 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.841486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.841559 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.841662 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.843131 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.844009 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.845621 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.846086 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.846228 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.850759 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.850764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.850768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.850800 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.853921 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.855452 (+ 1531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.855457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.855459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.855587 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.855977 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.855986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.855988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.856335 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.856608 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.856870 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.857105 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 01:35:07.860880 18468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: Bootstrap starting.
I20260925 01:35:07.862726 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:35:07.864461 18468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:07.865859 18468 log.cc:824] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: Log is configured to *not* fsync() on all Append() calls
I20260925 01:35:07.869089 18468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: No bootstrap required, opened a new log
I20260925 01:35:07.866088 18294 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37942
I20260925 01:35:07.879123 18319 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 4
I20260925 01:35:07.879968 18468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } }
I20260925 01:35:07.880402 18468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9ce18183314ee9bbb5517a6e74eeb6, State: Initialized, Role: FOLLOWER
I20260925 01:35:07.880931 18468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } }
I20260925 01:35:07.882038 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:07.882457 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:07.883179 18468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300106, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
I20260925 01:35:07.895394 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.879810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:41032)
0925 01:35:07.880015 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.880035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:07.880145 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:07.880491 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:07.880496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:07.880892 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:07.881053 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.881062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.882591 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.882596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.883508 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.883519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.884635 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.884641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.884856 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.885553 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:07.885571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:07.888073 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:07.891784 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.891789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.891793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.892341 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.892909 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.892926 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.892929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.893095 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.894362 (+ 1267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:07.894369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:07.894561 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 01:35:07.894695 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 01:35:07.895013 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 01:35:07.895383 18486 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:35:07.895895 18486 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:35:07.896683 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.879844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.27.160.60:33861)
0925 01:35:07.880054 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.880059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.880087 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:35:07.880250 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.880255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.880335 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.880461 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.881252 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.882431 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.883656 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.884487 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.884622 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.888291 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.888301 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.888313 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.888363 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.891605 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.892556 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.892563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.892567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.892749 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.893307 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.893343 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.893349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.894101 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.895717 (+ 1616us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.895947 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.896290 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:35:07.891718 18400 master_runner.cc:186] Error getting master registration for 127.27.160.62:43437: OK, instance_id { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" instance_seqno: 1790300106736219 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:35:07.902951 18319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84: Bootstrap starting.
I20260925 01:35:07.902998 18400 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:35:07.906354 18319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:07.907487 18319 log.cc:824] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84: Log is configured to *not* fsync() on all Append() calls
I20260925 01:35:07.910406 18319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84: No bootstrap required, opened a new log
I20260925 01:35:07.916121 18319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } }
I20260925 01:35:07.916503 18319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b61b9c36c4b4c9cac6ecb6591736f84, State: Initialized, Role: FOLLOWER
I20260925 01:35:07.916955 18319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } }
I20260925 01:35:07.917950 18489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:07.918334 18489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:07.919106 18319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:35:07.919310 18294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 01:35:07.919550 18294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:07.923035 18294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } }
I20260925 01:35:07.925688 18443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6"
I20260925 01:35:07.925868 18294 leader_election.cc:288] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [CANDIDATE]: Term 1 election: Requested vote from peers c1923c26647e4e56808bd86bf1c914b3 (127.27.160.61:33331), 0e9ce18183314ee9bbb5517a6e74eeb6 (127.27.160.60:33861)
I20260925 01:35:07.926230 18443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:07.929848 18443 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b61b9c36c4b4c9cac6ecb6591736f84 in term 1.
I20260925 01:35:07.930620 18270 leader_election.cc:302] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e9ce18183314ee9bbb5517a6e74eeb6, 5b61b9c36c4b4c9cac6ecb6591736f84; no voters:
I20260925 01:35:07.931183 18489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:35:07.941471 18489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [term 1 LEADER]: Becoming Leader. State: Replica: 5b61b9c36c4b4c9cac6ecb6591736f84, State: Running, Role: LEADER
I20260925 01:35:07.942256 18489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } }
I20260925 01:35:07.946995 18498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b61b9c36c4b4c9cac6ecb6591736f84. Latest consensus state: current_term: 1 leader_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:07.947489 18498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:07.951866 18500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:35:07.964738 18500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:35:07.973619 18393 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } attempt: 5
W20260925 01:35:07.976011 18273 tablet.cc:2369] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:35:07.983172 18443 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [term 1 FOLLOWER]: Refusing update from remote peer 5b61b9c36c4b4c9cac6ecb6591736f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:35:07.984838 18498 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 25 01:35:07 dist-test-slave-h6hf krb5kdc[18239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
I20260925 01:35:07.996395 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.974394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:33861 (local address 127.0.0.1:41044)
0925 01:35:07.974884 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:07.974904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:07.975028 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:07.976104 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:07.976112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:07.976769 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:07.977061 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.977079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.979029 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.979039 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.980882 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:07.980906 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.982531 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.982540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:07.982845 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.983642 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:07.983667 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:07.987387 (+ 3720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:07.992618 (+ 5231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.992623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.992628 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.993380 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.993944 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:07.993951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:07.993958 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:07.994155 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:07.995027 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:07.995035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:07.995256 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 01:35:07.995425 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 01:35:07.995816 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350}
I20260925 01:35:07.997816 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:07.975255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.27.160.60:33861)
0925 01:35:07.975496 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:07.975503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:07.975532 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:35:07.975622 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:07.975631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:07.975728 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:07.975852 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:07.977336 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.978813 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.981113 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:07.982100 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:07.982269 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:07.987654 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:07.987664 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:07.987672 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:07.987725 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:07.992416 (+ 4691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.993561 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.993574 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.993580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.993785 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:07.994316 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:07.994323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:07.994334 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:07.994786 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:07.994954 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:07.996981 (+ 2027us) server_negotiation.cc:299] Negotiation successful
0925 01:35:07.997324 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:35:08.006510 18393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3: Bootstrap starting.
I20260925 01:35:08.012002 18393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:08.024585 18393 log.cc:824] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3: Log is configured to *not* fsync() on all Append() calls
W20260925 01:35:08.031104 18421 tablet.cc:2369] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:35:08.042076 18393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3: No bootstrap required, opened a new log
I20260925 01:35:08.046459 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b61b9c36c4b4c9cac6ecb6591736f84. Latest consensus state: current_term: 1 leader_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:08.046833 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:08.040158 18252 master_runner.cc:186] Error getting master registration for 127.27.160.61:33331: OK, instance_id { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" instance_seqno: 1790300107247495 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:35:08.055801 18252 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:35:08.058820 18393 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } }
I20260925 01:35:08.059376 18393 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1923c26647e4e56808bd86bf1c914b3, State: Initialized, Role: FOLLOWER
I20260925 01:35:08.060050 18393 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } }
I20260925 01:35:08.067318 18518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:08.069815 18518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:08.069710 18368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c1923c26647e4e56808bd86bf1c914b3"
I20260925 01:35:08.069650 18393 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:35:08.070245 18368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:08.073544 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:08.073824 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:08.076177 18368 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b61b9c36c4b4c9cac6ecb6591736f84 in term 1.
I20260925 01:35:08.077220 18367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [term 1 FOLLOWER]: Refusing update from remote peer 5b61b9c36c4b4c9cac6ecb6591736f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:35:08.078879 18500 catalog_manager.cc:1347] Generated new cluster ID: b32e9200f35f43b7a9b1370c4ff4856d
I20260925 01:35:08.078976 18500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:35:08.079258 18489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:08.079583 18489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:08.079747 18489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:35:08.099809 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:08.014942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.27.160.62:43437)
0925 01:35:08.015229 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.015238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.026677 (+ 11439us) server_negotiation.cc:407] Connection header received
0925 01:35:08.027303 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.027314 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.027431 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.027526 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.031229 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:08.032838 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:08.050574 (+ 17736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:08.051466 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:08.051635 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:08.058636 (+ 7001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:08.058645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:08.058649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:08.058697 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:08.063520 (+ 4823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:08.066620 (+ 3100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:08.066630 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:08.066668 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:08.067105 (+ 437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:08.097526 (+ 30421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:08.097536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:08.097543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:08.098074 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:08.098367 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:08.099091 (+ 724us) server_negotiation.cc:299] Negotiation successful
0925 01:35:08.099440 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:35:08.120728 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:08.022926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.27.160.61:33331)
0925 01:35:08.023203 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.023213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.057017 (+ 33804us) server_negotiation.cc:407] Connection header received
0925 01:35:08.062183 (+ 5166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.062193 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.062296 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.062377 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.065691 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:08.067115 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:08.077663 (+ 10548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:08.078407 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:08.078546 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:08.106618 (+ 28072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:08.106626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:08.106631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:08.106676 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:08.111097 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:08.112074 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:08.112081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:08.112084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:08.112260 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:08.118109 (+ 5849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:08.118115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:08.118119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:08.118620 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:08.118890 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:08.119982 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0925 01:35:08.120315 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:35:08.125485 18471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:08.030928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.27.160.60:33861)
0925 01:35:08.034440 (+ 3512us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.034451 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.061847 (+ 27396us) server_negotiation.cc:407] Connection header received
0925 01:35:08.064849 (+ 3002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.064858 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.064961 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.065035 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.094832 (+ 29797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:08.096007 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:08.099742 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:08.100489 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:08.100608 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:08.112509 (+ 11901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:08.112528 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:08.112533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:08.112579 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:08.116567 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:08.117480 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:08.117486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:08.117490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:08.117650 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:08.122813 (+ 5163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:08.122820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:08.122823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:08.123262 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:08.123543 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:08.124717 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0925 01:35:08.125031 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3330}
I20260925 01:35:08.127408 18518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b61b9c36c4b4c9cac6ecb6591736f84. Latest consensus state: current_term: 1 leader_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:08.127768 18518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:08.127586 18529 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:08.127941 18529 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:08.137210 18521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b61b9c36c4b4c9cac6ecb6591736f84" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 43437 } } peers { permanent_uuid: "c1923c26647e4e56808bd86bf1c914b3" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 33331 } } peers { permanent_uuid: "0e9ce18183314ee9bbb5517a6e74eeb6" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 33861 } } }
I20260925 01:35:08.137820 18521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84 [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:08.156469 18534 catalog_manager.cc:1259] Loaded cluster ID: b32e9200f35f43b7a9b1370c4ff4856d
I20260925 01:35:08.156740 18534 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3: loading cluster ID for follower catalog manager: success
I20260925 01:35:08.159518 18500 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:35:08.160817 18534 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3: acquiring CA information for follower catalog manager: success
I20260925 01:35:08.161348 18500 catalog_manager.cc:1504] Loading token signing keys...
W20260925 01:35:08.162436 18534 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260925 01:35:08.183799 18500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b61b9c36c4b4c9cac6ecb6591736f84: Generated new TSK 0
I20260925 01:35:08.185230 18500 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:35:08.318989 18472 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.307567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.27.160.62:43437)
0925 01:35:08.307834 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.307843 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.316244 (+ 8401us) server_negotiation.cc:407] Connection header received
0925 01:35:08.317309 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.317315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.317416 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.317494 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.318739 (+ 1245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37962: BlockingRecv error: recv got EOF from 127.0.0.1:37962 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 01:35:08.321583 18471 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.307194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.27.160.60:33861)
0925 01:35:08.307380 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.307387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.319809 (+ 12422us) server_negotiation.cc:407] Connection header received
0925 01:35:08.320078 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.320085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.320175 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.320247 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.321300 (+ 1053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41052: BlockingRecv error: recv got EOF from 127.0.0.1:41052 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 01:35:08.322672 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.307265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.27.160.61:33331)
0925 01:35:08.307583 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.307590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.319455 (+ 11865us) server_negotiation.cc:407] Connection header received
0925 01:35:08.321086 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.321094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.321224 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.321304 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.322430 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52222: BlockingRecv error: recv got EOF from 127.0.0.1:52222 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
W20260925 01:35:08.364096 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.354627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.27.160.61:33331)
0925 01:35:08.354826 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.354833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.361742 (+ 6909us) server_negotiation.cc:407] Connection header received
0925 01:35:08.362332 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.362339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.362469 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.362549 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.363733 (+ 1184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52232: BlockingRecv error: recv got EOF from 127.0.0.1:52232 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 01:35:08.421372 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.404197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.27.160.61:33331)
0925 01:35:08.404377 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.404383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.418799 (+ 14416us) server_negotiation.cc:407] Connection header received
0925 01:35:08.419034 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.419041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.419154 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.419290 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.421101 (+ 1811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52240: BlockingRecv error: recv got EOF from 127.0.0.1:52240 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 01:35:08.475405 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.466937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.27.160.61:33331)
0925 01:35:08.467130 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.467137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.473486 (+ 6349us) server_negotiation.cc:407] Connection header received
0925 01:35:08.473811 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.473816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.473929 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.474015 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.475120 (+ 1105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52246: BlockingRecv error: recv got EOF from 127.0.0.1:52246 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 01:35:08.524510 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.513899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52260 (local address 127.27.160.61:33331)
0925 01:35:08.514250 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.514259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.522118 (+ 7859us) server_negotiation.cc:407] Connection header received
0925 01:35:08.522735 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.522742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.522857 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.522943 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.524207 (+ 1264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52260: BlockingRecv error: recv got EOF from 127.0.0.1:52260 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
W20260925 01:35:08.580737 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.571063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52262 (local address 127.27.160.61:33331)
0925 01:35:08.571316 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.571325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.578381 (+ 7056us) server_negotiation.cc:407] Connection header received
0925 01:35:08.578976 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.578990 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.579104 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.579196 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.580457 (+ 1261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52262: BlockingRecv error: recv got EOF from 127.0.0.1:52262 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260925 01:35:08.638156 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.635202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.27.160.61:33331)
0925 01:35:08.635425 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.635432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.636481 (+ 1049us) server_negotiation.cc:407] Connection header received
0925 01:35:08.636742 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.636747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.636850 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.636982 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.637861 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52274: BlockingRecv error: recv got EOF from 127.0.0.1:52274 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260925 01:35:08.677026 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.674523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52286 (local address 127.27.160.61:33331)
0925 01:35:08.674779 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.674789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.675404 (+ 615us) server_negotiation.cc:407] Connection header received
0925 01:35:08.675689 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.675697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.675776 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.675915 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.676732 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52286: BlockingRecv error: recv got EOF from 127.0.0.1:52286 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 01:35:08.756048 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.753641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.27.160.61:33331)
0925 01:35:08.753824 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.753830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.754412 (+ 582us) server_negotiation.cc:407] Connection header received
0925 01:35:08.754630 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.754638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.754754 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.754917 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.755779 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52296: BlockingRecv error: recv got EOF from 127.0.0.1:52296 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 01:35:08.889644 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:08.887494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.27.160.61:33331)
0925 01:35:08.887677 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:08.887682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:08.888148 (+ 466us) server_negotiation.cc:407] Connection header received
0925 01:35:08.888319 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:08.888324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:08.888397 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:08.888491 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:08.889345 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52300: BlockingRecv error: recv got EOF from 127.0.0.1:52300 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:35:08.897990 18486 catalog_manager.cc:1259] Loaded cluster ID: b32e9200f35f43b7a9b1370c4ff4856d
I20260925 01:35:08.898092 18486 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: loading cluster ID for follower catalog manager: success
I20260925 01:35:08.900599 18486 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: acquiring CA information for follower catalog manager: success
I20260925 01:35:08.902591 18486 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0e9ce18183314ee9bbb5517a6e74eeb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 01:35:09.153277 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:09.151196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.27.160.61:33331)
0925 01:35:09.151467 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:09.151473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:09.151755 (+ 282us) server_negotiation.cc:407] Connection header received
0925 01:35:09.151934 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:09.151937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:09.152021 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:09.152111 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:09.152925 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52312: BlockingRecv error: recv got EOF from 127.0.0.1:52312 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 01:35:09.165369 18534 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c1923c26647e4e56808bd86bf1c914b3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:35:09.677415 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:09.671163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.27.160.61:33331)
0925 01:35:09.671734 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:09.671741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:09.671896 (+ 155us) server_negotiation.cc:407] Connection header received
0925 01:35:09.672173 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:09.672177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:09.672343 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:09.672500 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:09.673031 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.673839 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.675197 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.675595 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.675734 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:09.676237 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:09.676754 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:09.676923 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:09.676984 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 01:35:09.677066 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":286,"threads_started":1}
I20260925 01:35:09.689752 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:09.683718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.27.160.60:33861)
0925 01:35:09.684181 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:09.684186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:09.684338 (+ 152us) server_negotiation.cc:407] Connection header received
0925 01:35:09.684484 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:09.684488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:09.684740 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:09.684893 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:09.685430 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.686137 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.687609 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.688006 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.688103 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:09.688548 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:09.689102 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:09.689285 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:09.689431 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 01:35:09.689500 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":259,"threads_started":1}
I20260925 01:35:09.717708 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:09.701763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.27.160.61:33331)
0925 01:35:09.701999 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:09.702016 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:09.709855 (+ 7839us) server_negotiation.cc:407] Connection header received
0925 01:35:09.710134 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:09.710143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:09.710293 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:09.710413 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:09.711315 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.712698 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.715011 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.715689 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.715837 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:09.716423 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:09.716566 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:09.716676 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:09.717271 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 01:35:09.717363 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:35:09.719203 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:09.702368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41076 (local address 127.27.160.60:33861)
0925 01:35:09.702619 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:09.702651 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:09.712214 (+ 9563us) server_negotiation.cc:407] Connection header received
0925 01:35:09.712628 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:09.712637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:09.712750 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:09.712856 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:09.713742 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.714790 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.716820 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.717425 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.717590 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:09.718241 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:09.718398 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:09.718505 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:09.718694 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 01:35:09.718869 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:35:09.721319 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:09.701970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.27.160.62:43437)
0925 01:35:09.711462 (+ 9492us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:09.711472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:09.712571 (+ 1099us) server_negotiation.cc:407] Connection header received
0925 01:35:09.713598 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:09.713605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:09.713842 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:09.714052 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:09.714936 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.715751 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.718352 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:09.719082 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:09.719200 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:09.720068 (+ 868us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:09.720596 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:09.720735 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:09.720881 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 01:35:09.720990 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9270,"thread_start_us":8257,"threads_started":1}
I20260925 01:35:09.727089 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18252
I20260925 01:35:09.728739 18314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:35:09.843165 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 18252
W20260925 01:35:09.865252 18344 connection.cc:535] server connection from 127.0.0.1:52174 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 01:35:09.865798 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18326
I20260925 01:35:09.866950 18388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:35:09.977058 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 18326
I20260925 01:35:09.998061 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18400
I20260925 01:35:09.999284 18463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:35:10.107447 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 18400
2026-09-25T01:35:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5235 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:35:10 dist-test-slave-h6hf krb5kdc[18623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:35:10 dist-test-slave-h6hf krb5kdc[18623](info): set up 2 sockets
Sep 25 01:35:10 dist-test-slave-h6hf krb5kdc[18623](info): commencing operation
krb5kdc: starting...
W20260925 01:35:11.282712 28288 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/test-user.keytab.
Sep 25 01:35:11 dist-test-slave-h6hf krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:35:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.160.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
Entry for principal kudu/127.27.160.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab.
I20260925 01:35:11.564605 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:36615
--webserver_interface=127.27.160.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:36615,127.27.160.61:42855,127.27.160.60:40721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:35:11.911916 18636 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:35:11.912359 18636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:11.912496 18636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:11.912653 18636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:11.920154 18636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:11.920310 18636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:11.920423 18636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:11.920516 18636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:11.920609 18636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:11.930871 18636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:36615,127.27.160.61:42855,127.27.160.60:40721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.62.keytab
--principal=kudu/127.27.160.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.62:36615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.160.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:11.932297 18636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:11.933718 18636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:35:11.946861 18641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:11.946774 18644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:11.946798 18642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:11.947676 18636 server_base.cc:1034] running on GCE node
I20260925 01:35:11.948606 18636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:11.950248 18636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:11.951521 18636 hybrid_clock.cc:648] HybridClock initialized: now 1790300111951460 us; error 41 us; skew 500 ppm
I20260925 01:35:11.951934 18636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:11 dist-test-slave-h6hf krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300111, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:11.960309 18636 init.cc:377] Logged in from keytab as kudu/127.27.160.62@KRBTEST.COM (short username kudu)
I20260925 01:35:11.962180 18636 webserver.cc:460] Webserver started at http://127.27.160.62:43961/ using document root <none> and password file <none>
I20260925 01:35:11.962963 18636 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:11.963106 18636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:11.963462 18636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:11.966692 18636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/data/instance:
uuid: "cb773a10452e4041b0d83b12c036afc5"
format_stamp: "Formatted at 2026-09-25 01:35:11 on dist-test-slave-h6hf"
server_key: "74e16ec6aaf9c0a0cf408aada63a88f7"
server_key_iv: "cbdf29beb4ba149769c76052123cfaf6"
server_key_version: "encryptionkey@0"
I20260925 01:35:11.967566 18636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/wal/instance:
uuid: "cb773a10452e4041b0d83b12c036afc5"
format_stamp: "Formatted at 2026-09-25 01:35:11 on dist-test-slave-h6hf"
server_key: "74e16ec6aaf9c0a0cf408aada63a88f7"
server_key_iv: "cbdf29beb4ba149769c76052123cfaf6"
server_key_version: "encryptionkey@0"
I20260925 01:35:11.973474 18636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 01:35:11.977341 18651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:11.978736 18636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:35:11.978955 18636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/wal
uuid: "cb773a10452e4041b0d83b12c036afc5"
format_stamp: "Formatted at 2026-09-25 01:35:11 on dist-test-slave-h6hf"
server_key: "74e16ec6aaf9c0a0cf408aada63a88f7"
server_key_iv: "cbdf29beb4ba149769c76052123cfaf6"
server_key_version: "encryptionkey@0"
I20260925 01:35:11.979195 18636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:11.995378 18636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:12.000347 18636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:12.000720 18636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:12.023020 18636 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.62:36615
I20260925 01:35:12.023039 18703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.62:36615 every 8 connection(s)
I20260925 01:35:12.024731 18636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/data/info.pb
I20260925 01:35:12.028923 18704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:12.031621 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 18636
I20260925 01:35:12.031821 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-0/wal/instance
I20260925 01:35:12.032380 28288 external_mini_cluster.cc:1428] Setting key 5ecb44ec80d3ea8ae56aa0878c10a2dd
I20260925 01:35:12.038491 18704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:12 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300111, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.61@KRBTEST.COM; defaulting to no policy
I20260925 01:35:12.064858 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.042620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:36615 (local address 127.0.0.1:59380)
0925 01:35:12.043525 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.043584 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:12.045738 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:12.046755 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:12.046760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:12.047331 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:12.047687 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:12.047707 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.050130 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.050133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:12.051274 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:12.051286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.052632 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.052637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:12.052831 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:12.053544 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:12.053572 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:12.055796 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:12.060538 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:12.060548 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:12.060558 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:12.060988 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:12.061570 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:12.061573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:12.061576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:12.061742 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:12.062797 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:12.062803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:12.063005 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 01:35:12.063227 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 01:35:12.063633 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":43,"thread_start_us":437,"threads_started":1}
I20260925 01:35:12.064908 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.042620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.27.160.62:36615)
0925 01:35:12.044384 (+ 1764us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:12.044393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:12.044448 (+ 55us) server_negotiation.cc:407] Connection header received
0925 01:35:12.046137 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:12.046148 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:12.046478 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:12.046693 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:12.047915 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.049959 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.051423 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.052412 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.052621 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:12.056045 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:12.056058 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:12.056077 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:12.056128 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:12.060368 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:12.061171 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:12.061184 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:12.061191 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:12.061445 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:12.061859 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:12.061864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:12.061868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:12.062649 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:12.062876 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:12.063409 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 01:35:12.063806 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":1537,"thread_start_us":1315,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.61@KRBTEST.COM" created.
I20260925 01:35:12.067888 18704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:35:12.069005 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.068617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37552)
0925 01:35:12.068785 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.068893 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 01:35:12.069289 18655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
I20260925 01:35:12.070907 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.070539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37564)
0925 01:35:12.070708 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.070775 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:35:12.071101 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:42855: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab.
Entry for principal kudu/127.27.160.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab.
I20260925 01:35:12.101698 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.61:42855
--webserver_interface=127.27.160.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab
--principal=kudu/127.27.160.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:36615,127.27.160.61:42855,127.27.160.60:40721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:35:12.120474 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } attempt: 1
I20260925 01:35:12.121882 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.121388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37572)
0925 01:35:12.121587 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.121754 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 01:35:12.122978 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.122639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37584)
0925 01:35:12.122782 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.122852 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 01:35:12.123248 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:42855: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
I20260925 01:35:12.191555 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } attempt: 2
I20260925 01:35:12.192842 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.192326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37590)
0925 01:35:12.192561 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.192701 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 01:35:12.193933 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.193553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37594)
0925 01:35:12.193731 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.193800 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 01:35:12.194101 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:42855: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
I20260925 01:35:12.285360 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } attempt: 3
I20260925 01:35:12.286718 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.286058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37596)
0925 01:35:12.286350 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.286492 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 01:35:12.288023 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.287491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37602)
0925 01:35:12.287697 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.287810 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 01:35:12.288283 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:42855: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
I20260925 01:35:12.431582 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } attempt: 4
I20260925 01:35:12.433012 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.432271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37604)
0925 01:35:12.432650 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.432783 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260925 01:35:12.434418 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.433917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37618)
0925 01:35:12.434114 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.434223 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 01:35:12.434779 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.61:42855: Network error: Client connection negotiation failed: client connection to 127.27.160.61:42855: connect: Connection refused (error 111)
W20260925 01:35:12.451911 18711 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:35:12.452389 18711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:12.452534 18711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:12.452684 18711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:12.460332 18711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:12.460500 18711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:12.460646 18711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:12.460748 18711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:12.460870 18711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:12.471486 18711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:36615,127.27.160.61:42855,127.27.160.60:40721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.61.keytab
--principal=kudu/127.27.160.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.61:42855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.160.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:12.472945 18711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:12.474473 18711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:35:12.487443 18719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:12.487568 18716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:12.487556 18717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:12.488524 18711 server_base.cc:1034] running on GCE node
I20260925 01:35:12.489418 18711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:12.490846 18711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:12.492112 18711 hybrid_clock.cc:648] HybridClock initialized: now 1790300112492031 us; error 42 us; skew 500 ppm
I20260925 01:35:12.492476 18711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:12 dist-test-slave-h6hf krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300112, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:12.495760 18711 init.cc:377] Logged in from keytab as kudu/127.27.160.61@KRBTEST.COM (short username kudu)
I20260925 01:35:12.497212 18711 webserver.cc:460] Webserver started at http://127.27.160.61:32993/ using document root <none> and password file <none>
I20260925 01:35:12.497915 18711 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:12.498044 18711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:12.498359 18711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:12.501282 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/data/instance:
uuid: "bf942bbfe1424be188fd9af7a1851b34"
format_stamp: "Formatted at 2026-09-25 01:35:12 on dist-test-slave-h6hf"
server_key: "dc3cd3122ebdc6f796464fd6e264172b"
server_key_iv: "542d60d0888986d29da6eb2d5f40608c"
server_key_version: "encryptionkey@0"
I20260925 01:35:12.502074 18711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/wal/instance:
uuid: "bf942bbfe1424be188fd9af7a1851b34"
format_stamp: "Formatted at 2026-09-25 01:35:12 on dist-test-slave-h6hf"
server_key: "dc3cd3122ebdc6f796464fd6e264172b"
server_key_iv: "542d60d0888986d29da6eb2d5f40608c"
server_key_version: "encryptionkey@0"
I20260925 01:35:12.507370 18711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:35:12.510984 18726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:12.512403 18711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:35:12.512598 18711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/wal
uuid: "bf942bbfe1424be188fd9af7a1851b34"
format_stamp: "Formatted at 2026-09-25 01:35:12 on dist-test-slave-h6hf"
server_key: "dc3cd3122ebdc6f796464fd6e264172b"
server_key_iv: "542d60d0888986d29da6eb2d5f40608c"
server_key_version: "encryptionkey@0"
I20260925 01:35:12.512842 18711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:12.531100 18711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:12.535866 18711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:12.536166 18711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:12.555366 18711 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.61:42855
I20260925 01:35:12.555389 18778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.61:42855 every 8 connection(s)
I20260925 01:35:12.556816 18711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/data/info.pb
I20260925 01:35:12.558943 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 18711
I20260925 01:35:12.559147 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-1/wal/instance
I20260925 01:35:12.559757 28288 external_mini_cluster.cc:1428] Setting key f616f9380497ecddbc6c65fcc84e3d01
I20260925 01:35:12.561823 18779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:12.572163 18779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:12 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300112, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.160.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.160.60@KRBTEST.COM" created.
I20260925 01:35:12.598626 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.575128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.27.160.62:36615)
0925 01:35:12.575708 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:12.575715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:12.576802 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 01:35:12.579931 (+ 3129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:12.579936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:12.580014 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:12.580117 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:12.582452 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.583348 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.586034 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.586639 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.586759 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:12.591509 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:12.591515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:12.591519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:12.591584 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:12.594782 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:12.595718 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:12.595727 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:12.595733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:12.595891 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:12.596340 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:12.596348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:12.596354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:12.596778 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:12.596939 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:12.597890 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 01:35:12.598152 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":283,"threads_started":1}
I20260925 01:35:12.599491 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.575729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:36615 (local address 127.0.0.1:59388)
0925 01:35:12.576571 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.576649 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:12.579574 (+ 2925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:12.580431 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:12.580446 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:12.581330 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:12.582202 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:12.582230 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.583518 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.583527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:12.585856 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:12.585875 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.586903 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.586910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:12.587273 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:12.588144 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:12.588187 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:12.591193 (+ 3006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:12.594966 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:12.594975 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:12.594991 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:12.595498 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:12.596007 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:12.596010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:12.596013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:12.596215 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:12.596914 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:12.596920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:12.597437 (+ 517us) client_negotiation.cc:769] Sending connection context
0925 01:35:12.597752 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 01:35:12.598174 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":319,"threads_started":1}
I20260925 01:35:12.601518 18779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:12 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300112, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:35:12.617745 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.602253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37628)
0925 01:35:12.602440 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.602456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:12.602546 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:12.603708 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:12.603712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:12.604175 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:12.604317 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:12.604326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.605895 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.605901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:12.606913 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:12.606926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.607939 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.607944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:12.608180 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:12.608790 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:12.608808 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:12.611154 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:12.614583 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:12.614591 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:12.614597 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:12.615138 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:12.615627 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:12.615631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:12.615635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:12.615826 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:12.616493 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:12.616500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:12.616693 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 01:35:12.616882 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 01:35:12.617268 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":42}
I20260925 01:35:12.617872 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.602555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.27.160.61:42855)
0925 01:35:12.603231 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:12.603238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:12.603263 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:35:12.603310 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:12.603317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:12.603499 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:12.603680 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:12.604508 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.605741 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.607180 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.607785 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.607924 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:12.611379 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:12.611391 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:12.611405 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:12.611433 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:12.614434 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:12.615321 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:12.615329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:12.615334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:12.615498 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:12.615964 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:12.615968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:12.615970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:12.616362 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:12.616623 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:12.617077 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 01:35:12.617449 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":526,"thread_start_us":374,"threads_started":1}
I20260925 01:35:12.620682 18779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:35:12.621973 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.621484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47748)
0925 01:35:12.621687 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.621792 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 01:35:12.622427 18727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
I20260925 01:35:12.623879 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.623526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47754)
0925 01:35:12.623681 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.623782 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 01:35:12.624128 18779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.160.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab.
Entry for principal kudu/127.27.160.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab.
I20260925 01:35:12.629362 28288 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskI60hYW/build/asan/bin/kudu
/tmp/dist-test-taskI60hYW/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.60:40721
--webserver_interface=127.27.160.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab
--principal=kudu/127.27.160.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.160.20:38789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.160.62:36615,127.27.160.61:42855,127.27.160.60:40721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:35:12.673488 18779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 1
I20260925 01:35:12.674681 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.674237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47762)
0925 01:35:12.674430 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.674535 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 01:35:12.675676 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.675341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47772)
0925 01:35:12.675505 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.675575 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 01:35:12.675916 18779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
I20260925 01:35:12.734045 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } attempt: 5
Sep 25 01:35:12 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300111, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:35:12.744160 18779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 2
I20260925 01:35:12.745316 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.744760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47788)
0925 01:35:12.744982 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.745100 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 01:35:12.746409 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.746004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47796)
0925 01:35:12.746159 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.746236 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 01:35:12.746655 18779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
I20260925 01:35:12.748337 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.734906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.27.160.61:42855)
0925 01:35:12.735110 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:12.735116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:12.735211 (+ 95us) server_negotiation.cc:407] Connection header received
0925 01:35:12.735413 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:12.735418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:12.735506 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:12.735623 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:12.736432 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.737199 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.738362 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.738823 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.738962 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:12.742306 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:12.742311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:12.742315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:12.742348 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:12.745511 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:12.746377 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:12.746382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:12.746385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:12.746551 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:12.747007 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:12.747013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:12.747017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:12.747292 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:12.747447 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:12.747830 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 01:35:12.748033 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:35:12.748410 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.734799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37640)
0925 01:35:12.735107 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.735128 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:12.735280 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:12.735657 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:12.735663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:12.736103 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:12.736279 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:12.736288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.737341 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.737346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:12.738207 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:12.738220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:12.738985 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:12.738991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:12.739211 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:12.739920 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:12.739938 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:12.742116 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:12.745681 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:12.745686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:12.745692 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:12.746246 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:12.746738 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:12.746744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:12.746747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:12.746895 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:12.747412 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:12.747417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:12.747581 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 01:35:12.747723 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 01:35:12.748034 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260925 01:35:12.749675 18704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:35:12.750780 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.750363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47802)
0925 01:35:12.750525 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.750661 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 01:35:12.751919 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.751519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47812)
0925 01:35:12.751733 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.751811 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 01:35:12.752200 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
I20260925 01:35:12.803478 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 1
I20260925 01:35:12.804867 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.804335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47818)
0925 01:35:12.804551 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.804698 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 01:35:12.806003 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.805666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47828)
0925 01:35:12.805828 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.805899 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:35:12.806277 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
I20260925 01:35:12.837920 18779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 3
I20260925 01:35:12.839207 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.838717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47832)
0925 01:35:12.838916 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.839027 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 01:35:12.840382 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.839948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47846)
0925 01:35:12.840187 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.840260 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260925 01:35:12.840674 18779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
I20260925 01:35:12.874511 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 2
I20260925 01:35:12.875828 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.875277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47858)
0925 01:35:12.875530 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.875661 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 01:35:12.876856 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.876529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47866)
0925 01:35:12.876689 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.876754 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:35:12.877044 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
W20260925 01:35:12.962445 18787 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:35:12.962852 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:35:12.962990 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:35:12.963122 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:35:12.970693 18787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:35:12.970820 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:35:12.970937 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:35:12.971020 18787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:35:12.971122 18787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:35:12.981284 18787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.160.20:38789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.160.62:36615,127.27.160.61:42855,127.27.160.60:40721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/krb5kdc/kudu_127.27.160.60.keytab
--principal=kudu/127.27.160.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.160.60:40721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.160.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:35:12.982836 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:35:12.983325 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 3
I20260925 01:35:12.983899 18779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 4
I20260925 01:35:12.984474 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:35:12.984648 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.984049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47870)
0925 01:35:12.984325 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.984434 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 01:35:12.984941 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.984433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47884)
0925 01:35:12.984655 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.984756 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 01:35:12.985863 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.985433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47898)
0925 01:35:12.985641 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.985728 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 01:35:12.985947 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:12.985604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47904)
0925 01:35:12.985745 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:12.985819 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 01:35:12.986122 18704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
W20260925 01:35:12.986166 18779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.160.60:40721: Network error: Client connection negotiation failed: client connection to 127.27.160.60:40721: connect: Connection refused (error 111)
W20260925 01:35:12.997195 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:12.997219 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:35:12.997277 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:35:12.997869 18787 server_base.cc:1034] running on GCE node
I20260925 01:35:12.998930 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:35:13.000574 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:35:13.001832 18787 hybrid_clock.cc:648] HybridClock initialized: now 1790300113001753 us; error 53 us; skew 500 ppm
I20260925 01:35:13.002170 18787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:35:13.005656 18787 init.cc:377] Logged in from keytab as kudu/127.27.160.60@KRBTEST.COM (short username kudu)
I20260925 01:35:13.007236 18787 webserver.cc:460] Webserver started at http://127.27.160.60:41327/ using document root <none> and password file <none>
I20260925 01:35:13.007951 18787 fs_manager.cc:359] Metadata directory not provided
I20260925 01:35:13.008039 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:35:13.008314 18787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:35:13.011549 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/data/instance:
uuid: "1d1343987b4c4d9a9975276bf4a510af"
format_stamp: "Formatted at 2026-09-25 01:35:13 on dist-test-slave-h6hf"
server_key: "84b4268b998a15939943a8a2907d3f2a"
server_key_iv: "3696919dd5b9d475633728afe1ea0800"
server_key_version: "encryptionkey@0"
I20260925 01:35:13.012329 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/wal/instance:
uuid: "1d1343987b4c4d9a9975276bf4a510af"
format_stamp: "Formatted at 2026-09-25 01:35:13 on dist-test-slave-h6hf"
server_key: "84b4268b998a15939943a8a2907d3f2a"
server_key_iv: "3696919dd5b9d475633728afe1ea0800"
server_key_version: "encryptionkey@0"
I20260925 01:35:13.017825 18787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:35:13.021404 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:35:13.023074 18787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:35:13.023283 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/data,/tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/wal
uuid: "1d1343987b4c4d9a9975276bf4a510af"
format_stamp: "Formatted at 2026-09-25 01:35:13 on dist-test-slave-h6hf"
server_key: "84b4268b998a15939943a8a2907d3f2a"
server_key_iv: "3696919dd5b9d475633728afe1ea0800"
server_key_version: "encryptionkey@0"
I20260925 01:35:13.023526 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:35:13.067696 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:35:13.072286 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:35:13.072578 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:35:13.091013 18787 rpc_server.cc:307] RPC server started. Bound to: 127.27.160.60:40721
I20260925 01:35:13.091020 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.160.60:40721 every 8 connection(s)
I20260925 01:35:13.092417 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/data/info.pb
I20260925 01:35:13.096131 28288 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskI60hYW/build/asan/bin/kudu as pid 18787
I20260925 01:35:13.096282 18854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:35:13.096328 28288 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskI60hYW/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790299737515179-28288-0/minicluster-data/master-2/wal/instance
I20260925 01:35:13.096901 28288 external_mini_cluster.cc:1428] Setting key ae9e0ca1b3a03fb9b3698288ba571500
I20260925 01:35:13.106603 18854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:35:13.121058 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.098758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.27.160.60:40721)
0925 01:35:13.099655 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.099677 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.099734 (+ 57us) server_negotiation.cc:407] Connection header received
0925 01:35:13.101519 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.101549 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.102097 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.102539 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.103907 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.105858 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.107155 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.108132 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.108301 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.112840 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.112847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.112858 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.112899 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.116363 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.117331 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.117343 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.117353 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.117544 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.117992 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.117996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.117998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.118550 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.118857 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.119151 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.119620 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":429,"threads_started":1}
I20260925 01:35:13.125034 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.109434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.62:36615 (local address 127.0.0.1:59400)
0925 01:35:13.110270 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:13.110303 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:13.110483 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:13.110880 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:13.110887 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:13.111488 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:13.111722 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.111730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.112773 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.112778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.113922 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.113935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.114973 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.114977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.115139 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.115745 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:13.115762 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:13.118331 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:13.122074 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.122085 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.122091 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.122715 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.123176 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.123180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.123182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.123342 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.123946 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:13.123953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:13.124130 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 01:35:13.124311 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 01:35:13.124623 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":496,"threads_started":1}
I20260925 01:35:13.125047 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.109447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.27.160.62:36615)
0925 01:35:13.109987 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.109995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.110406 (+ 411us) server_negotiation.cc:407] Connection header received
0925 01:35:13.110612 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.110617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.110702 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.110818 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.111871 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.112579 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.114082 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.114773 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.114915 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.118519 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.118525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.118530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.118568 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.121944 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.122845 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.122853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.122856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.123023 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.123455 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.123460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.123463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.123818 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.123979 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.124421 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.124630 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":239,"threads_started":1}
I20260925 01:35:13.126791 18854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.62@KRBTEST.COM
I20260925 01:35:13.140777 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.127476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.61:42855 (local address 127.0.0.1:37654)
0925 01:35:13.127682 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:13.127697 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:13.127768 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:13.128161 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:13.128165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:13.128760 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:13.128940 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.128952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.130083 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.130086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.131062 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.131075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.131827 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.131831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.131958 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.132636 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:13.132663 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:13.134927 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:13.138358 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.138365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.138369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.138742 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.139142 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.139147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.139150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.139300 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.139817 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:13.139822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:13.139973 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 01:35:13.140127 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 01:35:13.140410 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260925 01:35:13.140807 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.127530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.27.160.61:42855)
0925 01:35:13.127772 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.127780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.127809 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:35:13.127883 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.127891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.127988 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.128153 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.129171 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.129933 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.131203 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.131679 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.131813 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.135157 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.135162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.135166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.135199 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.138223 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.138870 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.138882 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.138887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.139019 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.139408 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.139416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.139419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.139696 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.139853 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.140223 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.140426 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:35:13.141964 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.124671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.27.160.62:36615)
0925 01:35:13.125375 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.125380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.125395 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:35:13.125449 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.125454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.125535 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.125590 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.128729 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.129644 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.130973 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.131474 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.131565 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.136258 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.136263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.136266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.136296 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.138954 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.139609 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.139616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.139619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.139759 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.140126 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.140131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.140134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.140395 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.140668 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.141457 (+ 789us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.141645 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":4178,"threads_started":1}
I20260925 01:35:13.141893 18854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300113, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.60@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.160.61@KRBTEST.COM
I20260925 01:35:13.156785 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.142684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47926 (local address 127.27.160.60:40721)
0925 01:35:13.142861 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.142867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.143027 (+ 160us) server_negotiation.cc:407] Connection header received
0925 01:35:13.143205 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.143210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.143302 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.143406 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.144118 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.145217 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.146746 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.147368 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.147495 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.150749 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.150757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.150763 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.150803 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.154036 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.154793 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.154797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.154799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.154903 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.155290 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.155293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.155295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.155679 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.155788 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.156325 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.156520 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:35:13.157199 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.142697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47926)
0925 01:35:13.142951 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:13.142964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:13.143080 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:13.143422 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:13.143426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:13.143862 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:13.143990 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.143998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.145345 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.145349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.146605 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.146619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.147528 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.147533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.147689 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.148288 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:13.148306 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:13.150567 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:13.154162 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.154169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.154174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.154670 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.155013 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.155017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.155021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.155184 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.155814 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:13.155820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:13.156045 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 01:35:13.156245 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 01:35:13.156729 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":126}
I20260925 01:35:13.160123 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.143643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.27.160.61:42855)
0925 01:35:13.144022 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.144026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.144038 (+ 12us) server_negotiation.cc:407] Connection header received
0925 01:35:13.144072 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.144076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.144132 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.144201 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.145696 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.146791 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.148221 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.148622 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.148759 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.153824 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.153829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.153832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.153863 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.157413 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.158135 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.158141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.158144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.158332 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.158769 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.158773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.158776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.159043 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.159257 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.159675 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.159856 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260925 01:35:13.163132 18854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: Bootstrap starting.
I20260925 01:35:13.163343 18704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 4
I20260925 01:35:13.165313 28288 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:35:13.167351 18854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:13.168812 18854 log.cc:824] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300111, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.62@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
I20260925 01:35:13.167963 18679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cb773a10452e4041b0d83b12c036afc5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59414
I20260925 01:35:13.176695 18854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: No bootstrap required, opened a new log
I20260925 01:35:13.179307 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.163952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47928)
0925 01:35:13.164166 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:13.164194 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:13.164302 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:13.164787 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:13.164792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:13.165368 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:13.165573 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.165584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.166964 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.166968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.168221 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.168234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.169211 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.169217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.169463 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.170149 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:13.170166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:13.172559 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:13.176330 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.176336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.176343 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.176802 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.177278 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.177286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.177291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.177496 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.178200 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:13.178205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:13.178419 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 01:35:13.178586 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 01:35:13.178909 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 01:35:13.179360 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.164302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.27.160.60:40721)
0925 01:35:13.164490 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.164495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.164512 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:35:13.164563 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.164568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.164644 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.164753 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.165731 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.166799 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.168397 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.168944 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.169167 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.172784 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.172789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.172793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.172824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.176181 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.176955 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.176959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.176962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.177091 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.177638 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.177643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.177647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.178050 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.178216 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.178730 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.178973 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:35:13.184903 18704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5: Bootstrap starting.
I20260925 01:35:13.186692 18854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } }
I20260925 01:35:13.187114 18854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1343987b4c4d9a9975276bf4a510af, State: Initialized, Role: FOLLOWER
I20260925 01:35:13.187613 18854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } }
I20260925 01:35:13.188467 18704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:13.188681 18862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.189100 18862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:13.189738 18704 log.cc:824] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5: Log is configured to *not* fsync() on all Append() calls
I20260925 01:35:13.189798 18854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:35:13.193218 18704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5: No bootstrap required, opened a new log
I20260925 01:35:13.200809 18704 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } }
I20260925 01:35:13.201246 18704 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb773a10452e4041b0d83b12c036afc5, State: Initialized, Role: FOLLOWER
I20260925 01:35:13.201781 18704 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } }
I20260925 01:35:13.202755 18875 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.203161 18875 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:13.204174 18704 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:35:13.204216 18679 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 01:35:13.204402 18679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:13.208518 18679 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } }
I20260925 01:35:13.212822 18829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb773a10452e4041b0d83b12c036afc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d1343987b4c4d9a9975276bf4a510af"
I20260925 01:35:13.213498 18829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:13.203994 18787 master_runner.cc:186] Error getting master registration for 127.27.160.62:36615: OK, instance_id { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" instance_seqno: 1790300112016852 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:35:13.219663 18787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:35:13.220057 18829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb773a10452e4041b0d83b12c036afc5 in term 1.
I20260925 01:35:13.220729 18679 leader_election.cc:288] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [CANDIDATE]: Term 1 election: Requested vote from peers bf942bbfe1424be188fd9af7a1851b34 (127.27.160.61:42855), 1d1343987b4c4d9a9975276bf4a510af (127.27.160.60:40721)
I20260925 01:35:13.220937 18652 leader_election.cc:302] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1343987b4c4d9a9975276bf4a510af, cb773a10452e4041b0d83b12c036afc5; no voters:
I20260925 01:35:13.225708 18875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:35:13.232933 18875 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [term 1 LEADER]: Becoming Leader. State: Replica: cb773a10452e4041b0d83b12c036afc5, State: Running, Role: LEADER
I20260925 01:35:13.234663 18875 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } }
W20260925 01:35:13.241408 18874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:35:13.241557 18874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:35:13.247822 18881 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb773a10452e4041b0d83b12c036afc5. Latest consensus state: current_term: 1 leader_uuid: "cb773a10452e4041b0d83b12c036afc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.248356 18881 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: This master's current role is: LEADER
W20260925 01:35:13.254060 18658 tablet.cc:2369] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:35:13.258092 18890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:35:13.285375 18779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } attempt: 5
I20260925 01:35:13.286537 18890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 01:35:13 dist-test-slave-h6hf krb5kdc[18623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790300112, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.160.61@KRBTEST.COM for kudu/127.27.160.60@KRBTEST.COM
I20260925 01:35:13.307143 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.243306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.27.160.60:40721)
0925 01:35:13.243550 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.243559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.246933 (+ 3374us) server_negotiation.cc:407] Connection header received
0925 01:35:13.247259 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.247264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.247349 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.247464 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.255468 (+ 8004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.257134 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.264691 (+ 7557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.265780 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.265927 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.289324 (+ 23397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.289331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.289336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.289380 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.294359 (+ 4979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.297578 (+ 3219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.297584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.297588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.297794 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.302323 (+ 4529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.302329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.302333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.303078 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.303346 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.306448 (+ 3102us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.306687 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:35:13.309343 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.286038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.160.60:40721 (local address 127.0.0.1:47958)
0925 01:35:13.286456 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 01:35:13.286478 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:35:13.286599 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:35:13.289367 (+ 2768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:35:13.289372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:35:13.289919 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:35:13.290181 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.290198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.294226 (+ 4028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.294234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.295636 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:35:13.295654 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.296695 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.296702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:35:13.296939 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.297579 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:35:13.297614 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:35:13.301649 (+ 4035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:35:13.306324 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.306330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.306335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.306894 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.307373 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:35:13.307378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:35:13.307381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:35:13.307553 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:35:13.308230 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:35:13.308235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:35:13.308394 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 01:35:13.308549 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 01:35:13.308880 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260925 01:35:13.311120 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.286833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.27.160.60:40721)
0925 01:35:13.288979 (+ 2146us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.288985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.289009 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:35:13.289076 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.289082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.289205 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.289333 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.290786 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.294066 (+ 3280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.295815 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.296492 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.297297 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.301891 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.301898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.301902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.301941 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.306159 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.307060 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.307066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.307071 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.307235 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.307726 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.307732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.307736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.308091 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.309758 (+ 1667us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.310399 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.310664 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1993,"thread_start_us":1819,"threads_started":1}
I20260925 01:35:13.320053 18779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34: Bootstrap starting.
W20260925 01:35:13.324708 18808 tablet.cc:2369] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:35:13.328007 18779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34: Neither blocks nor log segments found. Creating new log.
I20260925 01:35:13.330252 18779 log.cc:824] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34: Log is configured to *not* fsync() on all Append() calls
I20260925 01:35:13.343020 18779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34: No bootstrap required, opened a new log
I20260925 01:35:13.343031 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.246143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.27.160.62:36615)
0925 01:35:13.246345 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.246351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.263815 (+ 17464us) server_negotiation.cc:407] Connection header received
0925 01:35:13.267469 (+ 3654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.267480 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.267606 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.267704 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.277311 (+ 9607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.278955 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.303487 (+ 24532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.304592 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.304707 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.322839 (+ 18132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.322858 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.322868 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.322934 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.328638 (+ 5704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.333290 (+ 4652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.333303 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.333310 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.333564 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.335520 (+ 1956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.335531 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.335537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.336125 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.336496 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.341412 (+ 4916us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.341811 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:35:13.360275 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.264045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37680 (local address 127.27.160.61:42855)
0925 01:35:13.264301 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.264309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.313557 (+ 49248us) server_negotiation.cc:407] Connection header received
0925 01:35:13.313889 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.313897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.314006 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.314097 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.319676 (+ 5579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.321302 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.344894 (+ 23592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.346044 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.346202 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.352510 (+ 6308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:35:13.352517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:35:13.352522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:35:13.352570 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:35:13.356477 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.357523 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.357530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.357534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.357711 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:35:13.358232 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:35:13.358237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:35:13.358240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:35:13.358632 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:35:13.358873 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.359550 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.359829 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260925 01:35:13.363234 18829 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [term 1 FOLLOWER]: Refusing update from remote peer cb773a10452e4041b0d83b12c036afc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:35:13.364389 18881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:35:13.353919 18636 master_runner.cc:186] Error getting master registration for 127.27.160.61:42855: OK, instance_id { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" instance_seqno: 1790300112549964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:35:13.383667 18636 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:35:13.389153 18779 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } }
I20260925 01:35:13.389945 18779 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf942bbfe1424be188fd9af7a1851b34, State: Initialized, Role: FOLLOWER
I20260925 01:35:13.390796 18779 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } }
I20260925 01:35:13.395551 18862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb773a10452e4041b0d83b12c036afc5. Latest consensus state: current_term: 1 leader_uuid: "cb773a10452e4041b0d83b12c036afc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.395932 18862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:13.400695 18901 mvcc.cc:205] Tried to move back new op lower bound from 7333069264289480704 to 7333069263828176896. Current Snapshot: MvccSnapshot[applied={T|T < 7333069264289480704}]
I20260925 01:35:13.402877 18862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb773a10452e4041b0d83b12c036afc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.403164 18862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:13.408265 18905 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.408773 18905 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:13.410763 18754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb773a10452e4041b0d83b12c036afc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf942bbfe1424be188fd9af7a1851b34"
I20260925 01:35:13.411204 18753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:35:13.411458 18779 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:35:13.415560 18753 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [term 1 FOLLOWER]: Refusing update from remote peer cb773a10452e4041b0d83b12c036afc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:35:13.419812 18754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb773a10452e4041b0d83b12c036afc5 in term 1.
I20260925 01:35:13.422408 18875 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:35:13.427559 18875 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb773a10452e4041b0d83b12c036afc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.427904 18875 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:13.431452 18890 catalog_manager.cc:1347] Generated new cluster ID: 1a2a548274494789a54e3ff53db8e400
I20260925 01:35:13.431588 18890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:35:13.440304 18903 mvcc.cc:205] Tried to move back new op lower bound from 7333069264289480704 to 7333069263828176896. Current Snapshot: MvccSnapshot[applied={T|T < 7333069264289480704 or (T in {7333069264289480704})}]
I20260925 01:35:13.470100 18905 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb773a10452e4041b0d83b12c036afc5. Latest consensus state: current_term: 1 leader_uuid: "cb773a10452e4041b0d83b12c036afc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.470543 18905 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:13.471050 18914 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb773a10452e4041b0d83b12c036afc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.471530 18914 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:35:13.505024 18881 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb773a10452e4041b0d83b12c036afc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb773a10452e4041b0d83b12c036afc5" member_type: VOTER last_known_addr { host: "127.27.160.62" port: 36615 } } peers { permanent_uuid: "bf942bbfe1424be188fd9af7a1851b34" member_type: VOTER last_known_addr { host: "127.27.160.61" port: 42855 } } peers { permanent_uuid: "1d1343987b4c4d9a9975276bf4a510af" member_type: VOTER last_known_addr { host: "127.27.160.60" port: 40721 } } }
I20260925 01:35:13.505436 18881 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5 [sys.catalog]: This master's current role is: LEADER
I20260925 01:35:13.515964 18890 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:35:13.518424 18890 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:35:13.552337 18922 catalog_manager.cc:1259] Loaded cluster ID: 1a2a548274494789a54e3ff53db8e400
I20260925 01:35:13.552451 18922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34: loading cluster ID for follower catalog manager: success
I20260925 01:35:13.554288 18890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb773a10452e4041b0d83b12c036afc5: Generated new TSK 0
I20260925 01:35:13.556456 18922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34: acquiring CA information for follower catalog manager: success
I20260925 01:35:13.569149 18922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf942bbfe1424be188fd9af7a1851b34: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:35:13.570389 18890 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:35:13.736927 18898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:13.723545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.27.160.60:40721)
0925 01:35:13.723884 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.723896 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.734504 (+ 10608us) server_negotiation.cc:407] Connection header received
0925 01:35:13.735076 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.735096 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.735221 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.735309 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.736623 (+ 1314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47964: BlockingRecv error: recv got EOF from 127.0.0.1:47964 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
W20260925 01:35:13.741103 18784 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:13.738202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.27.160.61:42855)
0925 01:35:13.738400 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.738405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.738882 (+ 477us) server_negotiation.cc:407] Connection header received
0925 01:35:13.739174 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.739184 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.739280 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.739360 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.740821 (+ 1461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37686: BlockingRecv error: recv got EOF from 127.0.0.1:37686 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 01:35:13.742880 18858 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:13.727993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.27.160.62:36615)
0925 01:35:13.728214 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.728221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.738056 (+ 9835us) server_negotiation.cc:407] Connection header received
0925 01:35:13.739589 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.739596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.739695 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.739779 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.742616 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59432: BlockingRecv error: recv got EOF from 127.0.0.1:59432 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:35:13.795962 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.780070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.27.160.61:42855)
0925 01:35:13.780308 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.780316 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.787497 (+ 7181us) server_negotiation.cc:407] Connection header received
0925 01:35:13.788114 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.788121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.788361 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.788520 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:13.789402 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.790594 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.792695 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.793400 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.793507 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.794360 (+ 853us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:13.795097 (+ 737us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:13.795238 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.795500 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.795600 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 01:35:13.844238 18898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:13.830919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.27.160.60:40721)
0925 01:35:13.831115 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.831122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.841821 (+ 10699us) server_negotiation.cc:407] Connection header received
0925 01:35:13.842477 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.842483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.842649 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.842734 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.843923 (+ 1189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47972: BlockingRecv error: recv got EOF from 127.0.0.1:47972 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:35:13.895745 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.880603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.27.160.61:42855)
0925 01:35:13.880851 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.880858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.888566 (+ 7708us) server_negotiation.cc:407] Connection header received
0925 01:35:13.889144 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.889152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.889257 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.889329 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:13.890074 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.891202 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.893333 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.894035 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.894147 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.894826 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:13.894991 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:13.895060 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.895315 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.895408 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 01:35:13.943828 18898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:13.930691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.27.160.60:40721)
0925 01:35:13.930886 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.930893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.941623 (+ 10730us) server_negotiation.cc:407] Connection header received
0925 01:35:13.942152 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.942158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.942266 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.942341 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:13.943518 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47978: BlockingRecv error: recv got EOF from 127.0.0.1:47978 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:35:13.999214 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:13.985566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.27.160.61:42855)
0925 01:35:13.985758 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:13.985765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:13.992705 (+ 6940us) server_negotiation.cc:407] Connection header received
0925 01:35:13.993248 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:13.993255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:13.993355 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:13.993425 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:13.994141 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.995231 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.997087 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:13.997698 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:13.997788 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:13.998346 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:13.998475 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:13.998538 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:13.998767 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 01:35:13.998856 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 01:35:14.010464 18898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:14.008212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.27.160.60:40721)
0925 01:35:14.008439 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.008448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.008969 (+ 521us) server_negotiation.cc:407] Connection header received
0925 01:35:14.009261 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.009266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.009357 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.009473 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:14.010283 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47980: BlockingRecv error: recv got EOF from 127.0.0.1:47980 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:35:14.024873 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.018526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.27.160.61:42855)
0925 01:35:14.018849 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.018859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.019404 (+ 545us) server_negotiation.cc:407] Connection header received
0925 01:35:14.019641 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.019646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.019717 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.019824 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.020246 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.021068 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.022642 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.023343 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.023450 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.024049 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.024181 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.024244 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.024456 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.024555 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
W20260925 01:35:14.050239 18898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:14.047736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47986 (local address 127.27.160.60:40721)
0925 01:35:14.047953 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.047959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.048523 (+ 564us) server_negotiation.cc:407] Connection header received
0925 01:35:14.048742 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.048749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.048847 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.048922 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:14.050005 (+ 1083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47986: BlockingRecv error: recv got EOF from 127.0.0.1:47986 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 01:35:14.069814 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.063068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.27.160.61:42855)
0925 01:35:14.063286 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.063293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.063638 (+ 345us) server_negotiation.cc:407] Connection header received
0925 01:35:14.063802 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.063809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.063921 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.064058 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.064422 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.065876 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.067434 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.068176 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.068332 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.068784 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.068950 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.069078 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.069300 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.069430 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 01:35:14.078161 18898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:14.076136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.27.160.60:40721)
0925 01:35:14.076366 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.076375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.076733 (+ 358us) server_negotiation.cc:407] Connection header received
0925 01:35:14.076948 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.076954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.077042 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.077164 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:14.077931 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47994: BlockingRecv error: recv got EOF from 127.0.0.1:47994 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 01:35:14.110606 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.102852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.27.160.61:42855)
0925 01:35:14.103036 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.103043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.103625 (+ 582us) server_negotiation.cc:407] Connection header received
0925 01:35:14.103846 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.103853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.104019 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.104143 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.104664 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.106143 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.108314 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.108984 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.109084 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.109792 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.109945 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.110043 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.110213 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.110312 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 01:35:14.125367 18898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:14.123233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.27.160.60:40721)
0925 01:35:14.123438 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.123445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.123832 (+ 387us) server_negotiation.cc:407] Connection header received
0925 01:35:14.124061 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.124065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.124143 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.124259 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:14.125083 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48010: BlockingRecv error: recv got EOF from 127.0.0.1:48010 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:35:14.171932 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.163500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.27.160.61:42855)
0925 01:35:14.163736 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.163743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.164303 (+ 560us) server_negotiation.cc:407] Connection header received
0925 01:35:14.164747 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.164756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.164915 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.165053 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.165564 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.167097 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.169563 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.170325 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.170497 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.170989 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.171163 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.171255 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.171478 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.171577 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 01:35:14.182401 18898 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:35:14.179776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.27.160.60:40721)
0925 01:35:14.179955 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.179960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.180509 (+ 549us) server_negotiation.cc:407] Connection header received
0925 01:35:14.180697 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.180701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.180778 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.180908 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:35:14.182088 (+ 1180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48020: BlockingRecv error: recv got EOF from 127.0.0.1:48020 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:35:14.243837 18874 catalog_manager.cc:1259] Loaded cluster ID: 1a2a548274494789a54e3ff53db8e400
I20260925 01:35:14.243965 18874 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: loading cluster ID for follower catalog manager: success
I20260925 01:35:14.246696 18874 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: acquiring CA information for follower catalog manager: success
I20260925 01:35:14.248955 18874 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d1343987b4c4d9a9975276bf4a510af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:35:14.256718 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.250745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.27.160.61:42855)
0925 01:35:14.251014 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.251022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.251338 (+ 316us) server_negotiation.cc:407] Connection header received
0925 01:35:14.251662 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.251666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.251743 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.251858 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.252360 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.253260 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.254699 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.255164 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.255313 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.255767 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.255907 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.256069 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.256231 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.256337 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 01:35:14.268924 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.263104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.27.160.60:40721)
0925 01:35:14.263358 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.263365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.263697 (+ 332us) server_negotiation.cc:407] Connection header received
0925 01:35:14.263878 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.263884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.264052 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.264190 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.264567 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.265405 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.266728 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.267150 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.267294 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.267795 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.268313 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.268479 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.268530 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.268615 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:35:14.283133 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.276614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.27.160.61:42855)
0925 01:35:14.276787 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.276792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.277343 (+ 551us) server_negotiation.cc:407] Connection header received
0925 01:35:14.277630 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.277638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.277738 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.277844 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.278443 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.279377 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.281111 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.281631 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.281765 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.282301 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.282479 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.282590 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.282724 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.282848 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:35:14.283234 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.275017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.27.160.60:40721)
0925 01:35:14.275255 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.275265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.276413 (+ 1148us) server_negotiation.cc:407] Connection header received
0925 01:35:14.276705 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.276711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.276796 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.276868 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.277689 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.278702 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.280944 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.281648 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.281772 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.282470 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.282644 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.282728 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.282904 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.282984 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:35:14.283910 19018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:35:14.274958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.27.160.62:36615)
0925 01:35:14.275685 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0925 01:35:14.275693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:35:14.276597 (+ 904us) server_negotiation.cc:407] Connection header received
0925 01:35:14.276766 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:35:14.276774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:35:14.277013 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:35:14.277285 (+ 272us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:35:14.278068 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.279059 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.281236 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:35:14.281883 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:35:14.282012 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:35:14.282656 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:35:14.283128 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:35:14.283250 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 01:35:14.283532 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 01:35:14.283639 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":442,"threads_started":1}
I20260925 01:35:14.289425 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18636
I20260925 01:35:14.290475 18699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:35:14.402583 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 18636
I20260925 01:35:14.425496 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18711
I20260925 01:35:14.426928 18774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:35:14.542587 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 18711
I20260925 01:35:14.564110 28288 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskI60hYW/build/asan/bin/kudu pid 18787
I20260925 01:35:14.565377 18849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:35:14.674844 28288 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskI60hYW/build/asan/bin/kudu with pid 18787
2026-09-25T01:35:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4569 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9805 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (377111 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST